Analyzing Loan Application Data using Python | DataHour by Eshan Tiwari

Key Takeaways

need to take while performing Eda so we would be taking a real data set and after this session um analytics with that he would be sharing with you all the python notebook the PowerPoint presentation and the data set so no need to right now jump into the code part I would be explaining things in detail and then you guys can have access to my notebook as well as the entire data set uh one more thing while taking this session I am making this basic assumption that you guys know basics of python so I would be walking you all through that how I have run a particular code but I expect you guys to know the python so that is a kind of prerequisite and what I'll be doing uh after every five to ten minutes I'll check the Q a section so please use that for asking any question and I would keep on uh answering this okay so let's can you please enable a screenshot so that I can share the notebook yeah just a second yeah just check once if it's if you can share it now yep I can see this can you guys quickly confirm if my screen is visible okay perfect awesome so before I go into the coding part and like the technical Integrity let's understand the problem at a high level so I'm sure all of you or couple of you in the past have applied for credit cards loan and then generally Bank asks for a bunch of information and then the the there are multiple risk assessment model which basically help them to predict if I give a credit card or a loan to this person they are would they be able to repay the loan or would they be able to pay the credit card bills and if the field someone is at risk they don't give and if they feel it is a safe customer they give the loans now in this kind of exercise it is very important to develop a high quality model because if you don't give a loan to a good customer basically bank is missing out on the revenue however if you give it to a risky customer bank is actually missing out on Capital because they are going to make a loss so it's like a two-way spot and we need a model with a really high precision and recall and to develop these kind of models we need to spend enormous amount of time to understand what are the key features associated with a customer what kind of Trades I can pick up to predict whether this person this customer is going to be a good customer or not we look into historical data from the past and then on the basis of couple of features we decide okay what are the features which are good what are the features which are not good so to understand the quality of these features and how these features are working with like our main decision we need to develop a proper exploratory data analysis to understand the quality of features see if we could merge some feature delete some information to go ahead with the categories of features which are most useful and this is is the primary use case of Eda and which we are going to talk in this loan case data case study I want to quickly uh yeah I can share the Google collab URL can you give me a second so right now uh one second I'm giving you all viewer access I am putting it in the chat so this is the link and later I'll uh we will share every detail with you in the mail so don't worry okay uh before we go into the analysis part the kind of data set which we have uh I want to give you an overview so we are going to have two primary data source here first is you have loan application data basically uh what kind of uh data we have associated with the loan so if you see my sheet there's a Target which says that whether we should give loan to a customer or not and then there are a couple of features so we want to understand through this exercise that what are the features which basically help us to understand whether we should give uh loan or not so that is the target column so we would be understanding mostly in front of that okay now let's move on so before starting um Eda I am going to import some libraries and I will be just going through some all the details so that people understand the code so if you see in this part I'm importing the library so first there's importing warnings so python there are different different versions of python coming sometimes they throw some warnings which is not always necessary so we are filtering and then as we would be doing a lot of analysis on this data we are using numpy and Panda libraries and for visualization we are using matplotlib and c-bot now in this part what I am doing I have uh stored the data file in my local I am giving the path location of my data set now in data frame uh you guys can see the command I have used used here is PD dot read CSV so my data is stored in the form of CSV so I'm using this command you can connect your database with a live SQL databases you can do some web crawling you can read from PDF to the data frame allow us the flexibility to use the code from multiple perspective and then I am here just getting using this command head which gives me the top five columns of my data set so now as you guys see uh I'm starting the serial number with zero so in Python um the serial number or the indexing doesn't start from one it starts from zero so that is the number it is giving and that is here I am using you can see I have all the I can see the top level Columns of my data so want to quickly check if everyone clear with this and business statement has asked who are the customers here merchants or end user so uh bhupinder for this case study you can assume it's like an end user or retail uh users basically people like you and me who go to bank and apply for loans uh the reason I have uh you put 125 Max so it's completely up to you so for example I have here used the option head if I have just used app DS and did not use head command it will show me 125 rows so that's my personal preference you can make it 25 you can make it five you can make it 10. so there is no specific guidance on that okay uh then what does set option do the set options basically help me to Define how my layout would be looking what is the maximum number of columns I want to show maximum number of rows so if you guys see this section as I'm scrolling uh you guys can see that what are the maximum number of rows and columns I'm allowing to see if I set like maximum column as 100 then I would only be seeing 100 things so it basically helps me to set the UI part like what what kind of data cut from my data frame I wanna see okay now uh unlike Excel where you can visually see all the data or count rows python you can't do that intuitively or just by visual observation so there are bunch of commands which we have to use to see the data so to understand the data there are a couple of things we are going to learn first is this command so if you guys see application file when I'm getting a return reading from CSV I am putting it in a data frame called app.df so if you have to think of a data frame it is more like a tabular data set or SQL table similarly in Python we call it data frames so I'm calling my database as I have underscore DF now the First Command which is DF dot shape so it gives me like a uh it gives me uh this data set so the first element in the set tells me how many rows I have in this data set so it is telling that you have three lakh 7511 rows and how many features I have in this data set or features or you can call column in the machine learning world we call it more like a feature whereas in data we just call it column it tells me it's 122 column so even before without like counting how many columns we have how many rows this command help me to quickly understand the shape okay CSV file I'll share at the end of the session I don't want to create more restrictions so wait uh we would send the mail there would be everything or I'll put a link here okay now let's talk about understanding this data more in detail so now shape dot command generally helps me to understand sorry help me to understand how many rows and columns are there but maybe I'm more Curious I want to understand how many numerical data is there how many struct categorical data is there do I have any date column I just wanted to get a sense of how my data set looks what are the various columns uh data type so then you use this command called appdf with my data same name and then I use dot d types so dot details give me a list of all the columns along with what is the kind of data stored into this if you guys see skid current is integer Target is integer then bunch of object float all these things so this gives me a very nice summary of what is the various kind of data stored in each column uh so what happens like right now this row is a lot but sometimes we can also do a visual observation if we feel some data is stored in the incorrect manner we can even take the corrective method on the basis of this now moving on uh if you want like some more information about data without even looking so you can use this thing called appdf dot info so see it gives you more information it's first detail that you have these many entries three zero seven five one uh index from 0 to this then you can see there are 122 entries uh First Column of your data set is skid current and last column is amount required credit bureau yes then you can see that even there are 65 kind of floats 41 kind of integers and what is how many types are object or categorical data so this gives you a high level summary of your data set so if you guys see in the shape what we did we just understand about the sorry in the shape command we just understood understood about number of rows and column details give me a detailed about or uh all the columns in my data frame and then if I just want a quick summary of what is happening with my data frame it gives me now as you guys can see there are a lot of numerical columns and I want to just get a sense of like how the data distribution is happening in my numerical columns what is the mean Max median so you guys can also use this command called dot describe so what it does it looks into all the numerical columns and give you certain parameters like what is the count what is mean standard deviation what is the minimum value of 25 percentile 50 25 Max so it gives you some time to get uh understanding if there could be an outlier if there are data very spread along so you can get like a initial gut check of what how your data is look looking like okay so before I move to the next section let me quickly take some questions uh [Music] I'm just closing out on the questions ah so Muhammad Hassan has asked is it necessary to understand all the columns in our data set so uh Muhammad I think in real world depending upon uh what's your experience generally you might be working on a particular kind of Department in your company so you anyways get a good understanding of what is happening with your data set you know about most of the important columns however if you're taking data set from kaggle then generally there are some cheat sheet available what does each column mean so understanding is required so in a nutshell yes you mostly need to understand all the columns but if you are working professional working on the same set of data for quite some time then you have a already developed a sense however if you are picking data from external Source like kaggle then you have to go and read the documentation has been interpreted by pandas or we Define it so bhupinder it is generally interpreted by pandas and most importantly whatever is the data uh how the data is stored in your source so for example in my CSV let's talk hypothetically about date category if date category this is stored as a string then your pandas will also recognize this as a history so it interface but it interprets but it also depends on what kind how your data is stored in the source is asking which line of code I can use if I want to like to retrieve the format like dot here so uh what I'll recommend here generally if you uh you can do some search but it is PD dot read csb read uh PDF or something so along with read there are couple of uh so different different Source you can just uh change it so if you go to the official python documentation It is Well listed there so you have to just figure out from there okay is this kind of data set should so uh is saying is this kind of data set we should remove outlines lately I'll cover the section later in my discussion now utkarsh have asked what is the use of uh get I python earlier I was using Jupiter notebook uh now I am shifted to Google collab so in Jupiter there were some heat Maps where the alignment doesn't look good so this get Python and run line magic helps me to give a better alignment of my Legends and numerical data within my graph uh should we so guys how to uh to take care of data like removing outliers from removing duplicates I'll talk about it later can we see any findings or observation from the described query so update let's just uh look into some data sets for example uh can you see this target so now the target has value of 0.08 the max value of Target is 1 and other value is zero so it means that Target is saying that whether someone would uh should we give them loan or not then you can see that this is more like a Boolean it is not exactly a numerical data and only eight percent of the value has value one rest everything is zero so it helps me to develop an understanding like this similarly if you see count children now if you see majority of the people have like less than one children but then there is an outlier called 19. so this gives me an understanding that we need to treat outliers that is the kind of insights you can get from the describe guys null value delete duplicate outlier I'll cover all those things all these things later just have some patience uh so because like I said if you are working because it's asking the column names are not descriptive how should we analyze how should we interpret the or understand the importance so because what happened like I said if you're working in that industry for quite some time you already have a good understanding of what is the kind of data set and if you are new then generally there is a column data set provided which tells you about an overview of each uh the column and then we do a lot of analysis to understand whether this help us to develop some strong relationship with the target variable or not I will see it in a while so guys in interest of time quickly moving now before doing any Eda in the first step we download we imported the data then we got our basic field but now before doing any um Ada we also need to understand is my data good to treat for example if there are a lot of outliers missing values then my data is very noisy it might not give me right signal so before doing any analysis or even feeding in your uh model you need to do a data treatment so we do the data quality check and we look into missing value so in this section we are going to read about it so firstly uh most important problem which we see with most of the data set is high percentage of missing values so these are the columns which you want to treat so now what I am doing in this section 3.1 I am looking into all the columns which I have the 122 columns and I'm trying to see how many of these columns have some missing value so the command you use here is you start with looking into the data frame and then you use this command dot is null so if it is counting whether if there is any value which is null and then you take me so mean helps to understand so out of uh all the columns you have sorry all the rows you have in a column either you will have null or you won't have a null so you take 0 1 and it basically takes an average the average tells you what percentage of the columns in your data set has some missing value so we are looking into this data frame and then for each column creating a column called percentage of missing value and sorting it by missing value here so as you can guys can see initially we are not seeing any missing value not seeing any missing value because as you go deep at this point of time we start seeing some missing value amount of good price then we go down and then we keep on seeing and you can see some of the column have as high as 69 missing value so now we know that okay not our entire data CS set is clean somewhere it is missing value and we have to rate it so we'll talk about how to treat it uh any questions here uh can we put this project in resume completely up to you if um you can practice this data set and put it into your resume or if you have any git portfolio you can do that it is present in data and I'll also share this data set later please have completions okay so let's get started so now guys there is a rule of thumb for uh Missing value so if you find any column which have like more than 40 or 50 missing value then you know that half of the data is just blank in the best case scenario you are only getting half meaningful information which is not giving you any good Insight so there's no point using that data set so generally if we find any column which is like some in some business model people take 50 test threshold in some business model people take 40 percent as threshold so we basically dropped off with those columns if more than 40 or 50 we drop drop this column depending upon the industry if your data set has somewhere between 10 to 40 missing values then you think about two techniques you either do imputation or you uh drop the column now imputation is if you can impute it with some logical value so for example I'll give you a scenario if you guys ever go to a mall you will see there are people asking for surveys and in a lot of questions you these are not mandatory questions so you can actually Mark any not applicable or you don't Mark anything now if you know if people are not mentioning it and these are not applicable maybe this column is not very useful so you can drop the column similarly uh for example if in a survey they are asking you this question are you married your answer could be yes or no now depending upon your answer like in most likelihood if answer is no then there is a if they they won't ask you this question do you have kids so they can Auto assume things that it is not applicable or no need to track this answer now when you look into this kind of data and Survey it will look as a missing value but you know if you are not married then there's a very high likelihood you don't have kid so you can actually remove that not applicable value by zero similarly some of the data might be around time series so for example sales of data or sales of ice cream year on year so you can know that what kind of pattern does this data follow and if you find a missing value between two dates you can take average or medium so there are very various kind of imputation technique you can think of now if you find your missing value doesn't have a clear pattern it's very random then you should just delete the column because it's not going to give you any meaningful information now if you have less than 10 percent data then scenario slightly change you either impute the missing data in the rows uh imputation principle would be very simple or you just delete those rows because it's still less than 10 percent data is missing right apart from that 90 is there and it is Meaningful so we can think about retaining that data set so if you guys see in this command here what I have done I created this column if you guys see here percentage I'm calling this column percentage missing value so for each columns sorry yeah for each column here what I have done I checked whether if there are zero missing values I am calling it columns without no missing value so if you guys see 47.5 percent of the column 47.54 of the columns don't have any missing value if your data offset has greater than zero missing values but less than 10 then I am counting what percentage is this and I'm calling it columns having missing values between 0 to 10 so if you see 5.74 percent of my data has missing values similarly for 10 to 50 and further I have created this kind of data set so that is first that is the first step to checking whether your column has missing values or not once you get it then only you will think about uh analyzing the data uh quickly checking if there is any question uh Gobind I have not missed your question but I think that that is not likely related to the case study so what I'll do I'll take it in the end I don't want to create this section but in the end I'll uh take this question okay now uh so the way we have looked into the columns uh we are also going into the row the purpose of looking into row is uh like we have 122 data points per row and if we see there are some rules which have way too many missing values then it might not be useful so we would like to get a sense that what are the number of missing values in a row and if we feel that the number is way too higher maybe we might just drop those rows so if you see I have used the same command as earlier only thing is uh earlier I have not used this command XS so by default access is taken as 0 now I am using the word access is equal to one one is counting the data on the horizontal lines and it basically tells me for each rows how many missing values are there okay now talking about how to treat the missing values first step is we will be removing the column with high percentage of missing values so think like any column with less than a bit greater than 50 percent we will just drop it uh no point retaining this data so if you see what I have done I have created another data frame from the data frame one and what I have done wherever missing values is more than 50 I'm just getting rid of uh that data set and just removing those and wherever the missing values is less than 50 I'm continuing with this so now if you see application df1 data has only 81 rows whereas if you remember the application underscore DF data set it has 122 columns so 122 columns 281 41 columns we have just dropped it because they have more than 50 missing value now I'm again checking uh in this code like how many missing values are left in each column now you can see that any column which has more than 50 missing value they are dropped okay uh so somebody has asked if I have more than 60 mixing value with delete so like I said threshold is somewhere between 40 to 50 percent depending upon the industry on which you're working you will drop any column which has more than 40 or 50 percent of the data so if your data set has like more than 60 of the data as missing value you should definitely drop it I have not talked about outliers give me some time I'll cover come to that section so uh venkat has asked a good question in treating the data should we be starting with missing value or outliers or scaling the data so see my recommendation is always start with missing value reason is sometimes you missing value can lead to a scenario where you are dropping the entire column in scaling the data or treating the outliers you don't drop the the column you might remove some rows or you can do some capping in the rows but you are not going to delete the entire column whereas in the missing value you can actually delete it so start with deleting the data and creating the missing value and then based on how the data set is looking you can take the further treatments okay guys please use the Q a section because I am not monitoring the chat okay now if you guys look into this data set there are some columns with 48 there are some column with 13 so I am now take taking a scenario where I'll be treating you how do I go about imputation so if you look into this data amount required credit bureau monthly weekly daily are all these columns have 13 missing value so I can I have a sense that they would be missing the same rows okay so now we will think about can we do some kind of imputation here so now if you look into uh this section 3.3.1 what I am doing I am looking into identifying the imputation technique for amount required credit bureau month now if I do a box plot I I'll cover the box plot in detail but box plot is a technique to look at the outlier value so so there are a lot of outliers so we have to figure out what's happening with the rest of the data and can we do some imputation so now if you guys look into these values what I am doing for each data point in the amount required credit bureau month section I am counting the number of values so 0 has 2 2 3 3 these many values one has these many values and so on it keeps on declining right so if I convert this into percentage so in this column 83 percent of your data set is having the value 0. and 12 of your data set has the value uh as one rest all is very less okay so now as you can see when I looked into the describe column for this data set up to 75th percentile all the values are zero and then we got some high maximum value which basically helps us to understand this is uh this has some outliers so now if I just analyze this data 83 percent of my data set has just one value zero so then it gives me some confidence I'm not saying we are going to be 100 correct then there is if I look into the data set 83 times zero value is coming so it gives me an understanding that 83 0 is the maximum occurring value so like a mod value of this data set so maybe I can think about replacing all my missing values with 0 and this technique is called mode imputation similarly if we started with this month data amount required credit bureau month now look into amount required credit bureau weak again we are seeing some missing value again the same technique which I applied looked into the percentage I found that 96 of the data set has only zero value so it gives me some confidence that I can do a mode imputation where I can replace all the values with 0. so so far I'll be continuing with these approach and keep on checking how the data sets in some cases it is 99 and all so where I see a significant of the data could be just replaced by one missing value I'll just replace it so that is how we do uh how uh Missing value treatment so uh govind has asked I work in healthcare research how do I decide percentage of drop of data um I'm not sure uh to go with it depends on the kind of features you are talking about so for example if you are working in uh let's say you are working on Diabetes Type 2. and the column which you are using uh it says that weight of person now if even 40 of the data set has missing value then I know that for 40 of my customers I can't use the weight and weight is a very important parameter for predicting diabetes right so you can then think about dropping the column because it would give you more fall signal however let's say uh again you are looking into this diabetes data and it has a column called your name right now your name has let's say even five percent or ten percent missing value but your name doesn't help you to predict whether you should you would be having a diabetes or not so even in five ten percent you will remove the data so uh you will drop the column with the missing value so you have to basically think what is the use case of this data and if you have way too many missing values would it bias your result and then you can take a cut off threshold yes I don't know what do you mean by feature selection for it so guys there is a package called rfe so whenever you are building a regression model or anything you the more number of features you use uh model penalize your model so we use some techniques which help us to pick high quality feature so yes that might be a good technique but right now let's say if you don't have rfe model uh package available for you then you will do an exploratory data analysis to understand what are the features which are having important importance in respect to Target column okay someone has asked if this session will be only doing Eda yes the sessions name is analyzing the loan data and doing Eda so yes this is going to be an EDS session we are not doing any model development here so naresh has asked few features are also important when we draw features like greater than 50 missing values at what time will we do so naresh C uh if that feature is really important but if it is not available for 50 of the cases think you might be having a high probability of drawing a wrong inference like around 50 of your population doesn't even have that feature so I don't think that feature is important you should not be including it so bhagashi has asked what do you mean by mean mode median so badshi let's say you have to think about how to replace your data set how to replace missing value in your data set so if you feel there are some data set where mean is a standard deviation is very low so all the values are very close to me then you can replace all the missing value with mean value right similarly if you are looking into some data set we are like in similar to our case like if you see this uh section amount required credit bureau per day now 99.44 of the uh values in this column is zero right so it means that values which are even missing they have a very high likelihood of using zero this is called mode imputation somebody has half the zero means and missing value here no 0 doesn't mean a missing value it is basically somebody asks you how much amount is required in the credit bureau day or deposited in the credit credit bureau today and if that value is 0 it means there is no transaction happen it's not a missing value so guys if you go to this start of the data set like here you will see there is a column called Target so Target means whether you will uh Grant the loan or you don't so that is what our Target column tells us okay uh taking pause from questions and going again to uh going again to the analysis part so now if you guys have seen I have already talked about how I'll Place missing values removing those columns and after that how did we think about uh imputation we found that in our data set mode imputation is the right technique so we went ahead with this and replaced all the columns which has 13.50 missing values with mode imputation okay uh then I think in this piece of code I have just repeated the same thing so I'll quickly move to the next section now another thing which we have to do is fixing the data type so sometimes lot of our data is stored incorrectly for example uh your data could be Boolean like zero or one it is more close to a categorical column that a numerical column so if it is coming as integer you might be interested to change this similarly for example dates or sometimes even numbers are stored as a column or ear is stored as a numerical data so you might be interested to change that into categorical columns so good practices look into the column name column description and see if the proposed data type is the correct data type or not okay now we are moving to the next section which is identifying outliers for numerical data so lot of people have asked about outlier treatment why outlier treatment is important so I'll give you a very quick sense of why outlier treatment is important first of all so let's think about a hypothetical scenario let's say we all there are 10 of us who pay tax of like 1 lakh per day sorry one lakh a year so how much tax we would be giving one lakh on an average and 10 lakh total now by any chance of sampling error or something some extra genius factor in those 10 people you got uh Mukesh Ambani also so now the nine people would still be paying one lakh has a tax and mukesha money might be paying 100 crores so now if you take the first case the average tax is 1 lakh whereas the scenario where vocation money was added the average tax could be in crores so if government is not aware about this data set they might make an assumption oh we are getting like crores in text so we can do more Investments and more planning and in reality what happened there was an outlier like not everyone is going to earn same as Mukesh Ambani and that's why the tax we have projected it's going to be listed so that's why we have to remove the outliers now let's learn about the outlier removal technique so the first step to removing any outlier is to check whether it has an outlier or not the best technique to look at an outlier is through using box plot technique in the box plot technique if you guys see you will feel that there is this uh this line and then there is this box and then again there is this line and then bunch of dot so so if you guys see this is smaller line let me see if I can do some annotation if you guys see this smaller line this tells you minimum allowed value this line tells you maximum allowed value this one tells 70 5 percentile this line tell me 25 percentile this middle value is your median now all the data points which are lying here these are called outliers so this is more than maximum allowed value so once you draw a box plot on a feature and if you see something like this you know there is an outlier and you have to do outlier treatment okay let's quickly see if there are some questions here people is asking when we use median imputation so guys median imputation is used in the in place of uh mean imputation where your data has lot of outlier values the mean of median works in the same way it's just if there is outlier values uh your mean would be severely impacted by outlier so you will use median okay yes Gobind I do have a data dictionary in after class we will share all the data I think for median imputation I have already answered it uh so yes has asked something with outlier treatment he replaced it by mean mode median uh his values are still having outlier so yes what happened if you do uh if there's an outlier and you are doing a mean uh replacement it in the even in this that mean value your outlier has some effect it might show you make your mean very high and that's why I'll tell you the right technique to do this okay uh so let's quickly move on now if you guys see here what I have done I have calculated 25th percentile 50th percentile 75th percentile for my data set now for outlier removal there is a standard formula the maximum allowed value in your data set is 75th percentile so if you guys see here what I'm doing I'm calculating 75th percentile and then you calculate range so range is 75th percentile minus 25th percentile and then you take the 75th percentile here plus 1.5 times into the uh into the range so this tells me the maximum allowed value is one three four one and minimum allowed value uh for a box plot is uh minimum allowed value is q25 so 25th percentile minus 1.5 into range this is a mathematical formula which is derived I am not divising it so you can use the formula as it is and now uh if you guys see in this case I am not seeing any lower outlier because there is no dotted point I am talking about this if you say there is no our lower outline there's no data point outside this but there are a lot of upper outlier so what I'll do here if you look into the formula part here which I have written I am only doing an upper limit so I'm calculating the 75th percentile and 1.5 that's calculated the upper limit and in this step if you see what I have done I have looked into the data set and saw where how many outliers are there there are 4.79 percent outliers basically the data points in a column which is having more than allowed value and then what we can do we can replace all these maximum value with the upper outlier so you can cap it so for example if I found that uh if we are looking into the weight of people and we found that uh weight more than 100 kgs in outlier maximum allowed value is 95 so all the bit above 95 kg would be capped at 95 and that is a technique for outlier removal is there any question why we choose only 1.5 it is a mathematical formula I can explain but it will take more a big amount of time in the class so I can actually try to find some article so you guys can actually later read this article they have actually explained it in a much better manner I have put this into the chat section so you guys can have a look this is have all this have all the details similarly if you guys see the amount total income or amount credit all these have outlier values so I have used the same technique I have calculated none of these have lower outlier everyone has everything has upper outlier so I have kept the upper outlier values here okay let's quickly take uh so somebody has asked can you explain interquartile range so interquartile range is nothing you look into a data set you calculate the 75th percentile and 25th percentile and take a difference of that uh another I have already shared an article why we choose 1.5 you can read about it uh I'm saying this is the first time yeah thank you I'm thank you for uh I'm happy to share with you for the first time but yeah this is a very standard technique uh so going uh there are two School uh of thoughts so some people use streaming and some people use vincerization so trimming means any data set which has outlier we will remove that data whereas vincerization cap the data I prefer winterization because in my column uh or in my row if I'm dropping the outlier there could be some other features also which might be useful for better prediction now if I drop that particular row I'll be missing out on those features so unless you have a humongous data where you have like lot of rows available my recommendation is just go by vincerization which is capping the data rather than removing it uh yes when you say skewed data it primarily means it has an outlier so all all the things which we are talking about is skewed data lately I think if your data has 15 20 outlier I'm not sure if that would be a true outlier because you are talking about interquartile vein and then it could be only possible when there's a difference between 75 percentile at 50 100 percentile and it's a big difference across each percentile column and even in those scenario if these are outliers you can do a capping like a wincerization uh I'm coming to this okay guys we are about time so let me quickly move to some other sections so uh here I have again put more quotes about how I analyze all the other features so again we will be sharing these cheat with you all so you can read now let me quickly come to the exploratory data analysis part which we are here Eda so guys in our data analysis uh we if you guys can see here we have primarily two kinds of data first is numerical data and other is uh categorical data and for each of the kind of data set we will be doing three kinds of Eda first is called univariate so second is bivariate and third is multivariate Eda univariate is more like a histogram so for example if I got a column um called let's say gender now gender could be male or female so if I just try to find uh how many male and female are in my data set that is more like a univariate analysis Now by variate if I have to do where I compare two variables together so if I want to find uh in the list of defaulters what is the proportion of male to female versus in the list of approval what is the proportion of male to female so what I am doing I am doing basically taking two variables Target as well as gender and then I am doing uh sorry sorry I'm taking Target and gender so I'm comparing two variables together that is called bivariate analysis and then sometimes we use pivot table we have multiple features are compared together that's that is called a multivariate an answer so let me give quickly give you an example of this so if you guys see we have these categorical columns right and I'll tell you one example of a universe analysis so now if there is a column called uh name contract type now uh name contract type in the Target 0 what is the kind of loans people are having so when we are not granting the loan eighty percent of the loans are cash loans whereas the revolving loans are 20 percent similarly if we look into the allowed loans again more than 80 percent are cash loans and very small amount of revolving loans so there is not much too distinguish between uh the loan type for uh allowed customer as well as not allowed customers so this is not a good feature for us now we will try to see if there is any feature where we see a big distinction let me take some other example uh for example let's understand this for the people who are getting how uh who are getting loan versus not getting loan let's look into uh what kind of houses they are looking so people who did not get loan they ate more than 80 percent have houses whereas people who are getting loan uh they again they have more than 80 percent have houses a few of them are living with parents so if you guys see there's not much Insight in all the features so I have run a very similar analysis and then what I found these analysis so for the people who are uh defaulters generally the percentage of male is much higher similarly for the defaulters higher percentage of customers with secondary education again in the age group of defaulters there is a higher number of people who are within the 30s so these are the kind of feature analysis which you can do with uh uh uni variate or a bivariate analysis and it helps you to understand what are the features we should be caring about similarly if we have let's say we have bunch of numerical columns now my target the column which I care about is Target column now there could be some columns which are Auto correlated so again taking the example of diabetes if I have two data set weight and BMI so if you guys see if somebody somebody has a very high weight then BMI would be further higher for that person so whether I use the weight variable or BMI variable it would give me same Insight so we have to find what are the columns which have a strong correlation then we do a multivariate analysis like I have done here so for example if you see amount of good price have a very strong correlation with amount annuity so now I don't have to take both the features they have very strong correlation I can just take one of that so generally it comes very handy in the world of machine learning where you want to reduce the number of features in your data set okay does this make sense so guys this is how we perform multivariate analysis by variate analysis mostly bar graphs and all and then uh we can keep on doing analysis like that so I think we are about time so with this I'll take a pause and uh would give would give it back to the moderator uh yeah a launch a feedback poll just to get the feedback feedback of the session so you guys there was a lot to talk about I could not complete everything here yeah this is my LinkedIn profile if you guys wanna connect yeah uh I would like to uh thanks uh thank God lord and on the behalf of analytics Vidya I would like to thank you for uh your time ishan and for delivering such an excellent session I'm sure our audience spy founded insightful and hopefully we can conduct more such sessions with you in future I hope you guys have filled the feedback poll if not I request you to please fill in the uh poll uh about feedback as it help us to conduct more successions the recording of the of this session will be available within two two days on our YouTube channel I will post the link uh in the chat one more time good we will be uh back with another session of data around 27 that is tomorrow uh the link is in the chat uh section till then uh take care and keep learning