Outlier detection and removal using IQR | Feature engineering tutorial python # 4

Key Takeaways

in this tutorial we will be removing outliers using IQR technique as usual we'll go over some theory first then we'll write code in Python pandas to remove the outliers and in the end we'll have an interesting exercise for you to solve here I have a heights database now visual examination you can easily tell that there are some outliers here these heights are in feet of course you cannot have a person with 40 feet height hence these all of these are outliers you know anything more than 77 point 1 similarly this is an Adel data set hence one point to height is clearly these two heights are outlier now how can you detect the same thing using an IQR technique in order to understand IQR you first need to understand percentile now what is percentile well percentile is basically for example here I have this five point three file which is 25th percentile it is also called q1 25th percentile value means in your data set 25% of samples are below this value total samples here are twenty twenty-five percent of twenty is five and five samples here are having value less than five point three five you see 5.0 5.1 all these values are less than five point three five hence this is 25th percentile similarly 75th percentile will be a value which is six point twenty seven so 75% of samples are having value less than six point two seven I hope you get it so that's why six point two seven is called 75th percentile your maximum value will be called hundredth percentile because of course all the values are below that value similarly minimum value will be called zero percentile because that doesn't have any value less than that particular value right so this is a quick summary of percentile now once you know percentile you can easy calculate IQR by dipping q3 and q1 here q3 which is 75th percentile is 6.27 - 5.35 is q1 point nine to five is your IQR value now to remove outliers you need to come up with lower and upper limit your lower limit will be q1 which is 25th percentile - one point five IQR now IQR if you think about it is just if it is this distance you know it is this much distance so you take that distance multiplied by 1 point 5 so you add 1/2 into it and then you extend that from here so from 5 point 35 minus 1 point 5 and you get this value 3.96 similarly upper limit will be q3 plus 1.5 IQR you know they just came up with this number you can just remember this that you have to do 1.5 IQR plus q3 to get your upper limit once you know the upper and lower limit you can exclude audit all data points which has value less than a lower limit so 3.96 see these two are excluded there they are termed as an outlier and any value a bow upper limit which is 7 point 66 can also be excluded so you can clearly see that these two three values are now excluded so just by looking at this data set visually you can kind of say that IQR is kind of working okay in this case because most of the people will have heights in this range adult having height two point three feet is very very it's probably not not possible and adult having a height of 14 feet is also not possible the person doesn't need stairs if he has this much height and just go to your first or second floor all right let's get into coding now i have that height CSV here and I'm gonna load that into my notebook and it will look something like this so here is my data frame and now from this data frame I will quickly do DF no subscribe describe to see the statistics so you can see that when this is 25th percentile value 75th percentile value maximum is 4 T minimum is 1 point T I always use describe function because it can tell you those quick stats ok now let's calculate Q 1 and Q 2 now by the way you can get Q 1 and Q 2 from here as well but the other way of getting Q 1 and Q 2 is you take your height column and you say quantile quantile is a function so one time and if you want 25th percentile this is what you need to do and if you need 75th percentile this is how you do it and I will quickly print q1 and q3 so these are my percentage value IQR will be very simple formula guys this IQR is really a very very simple technique IQR value is this and now I can easily find my lower and upper limit using this formula so I'm doing q1 minus 1 point 5 QR q3 plus box 1.5 QR these are my upper and lower limit and once you have upper and lower limit in a data frame you can easily detect these outliers by doing this you can set height less than lower limit or height greater than upper limit those are my outliers and you can clearly see see all these are coming as outliers these are the same ones that we saw in our presentation the cells marked in the yellow color all right now if you want to remove these outliers from your original data frame what you can do is something like this so I'm using and condition I want to make sure the remaining samples are within this limit of lower limit and upper limit okay and the resultant data frame that you get you can maybe store that into a new data frame call it DFM outlier and this is how it looks alright so that's all I had now the most interesting part of this tutorial which is an exercise believe me I'm not giving you any hard assignment these are very simple assignments but it will consolidate your understanding once you practice it on your own what you're going to do is use the heights and weights data set it is very similar to what I use but it has height and weight to parameters I got this file from this giggling you can download it or if you don't want to download it you can just go to my github page and you will find it in the exercise folder and then you need to plot histograms first for weight and then for height and then use IQR to detect outliers based on weight and then do the same thing based on height now I have a solution link but I know you you all are very good students since the students so you're not going to click on this link until you have practiced this on your own all right thank you very much for watching I will see you in next feature engineering tutorial