Handling Missing Data - p.10 Data Analysis with Python and Pandas Tutorial

Key Takeaways

what is going on everybody Welcome to Part 10 of our data analysis with python and Panda tutorial Series in this part what we're going to be talking about is handling missing data you're going to see this usually as Nan or Nan which means not a number uh but generally all missing data will be called not a number regardless of whether or not it's anything at all and you basically you have a lot of choices when it comes to handling for missing data and each choice is valid in its own right in various uh circumstances but a lot of times the choice that you make to handle missing data could be the wrong one so there's a little bit of philosophy that goes into which option you may end up choosing so when you have missing data you basically have four major choices okay you could ignore the data you could just ignore the fact that there's missing data right you you totally right to do that uh then you could delete it okay so you could delete uh and really deleting it as there's two two options here but for now we'll just say delete it uh so remove the missing data okay uh then you've got fill in the missing data so you could take you know previous the previous data point and just kind of copy that into the the missing one right now uh you could take the future data point kind of push it backwards into the missing data point okay so that's filling um and then finally you can replace not a you know missing data with some sort of static number or something like that so those are your four major choices and we're going to be covering all four uh in this tutorial and as and as well as kind of covering why you might choose that specific option and when you maybe shouldn't use any of the options or maybe not do anything and maybe ignore it so the first option that we're going to cover is ignore it right uh that's really quick tutorial uh you do nothing so uh the next option would be uh to delete it so that's what we're going to cover uh second I suppose so what we're going to do uh instead of saying Texas one year equals this we're going to actually add this to the data frame so we're going to say um HPI data and we're going to redefine and call a new column we're going to say tx1 year like that that's fine then um we're not going to be we don't really need to plot anything at the moment necessarily um let me do this and and we're going to do this and we're going to say we want to print HPI data but we're going to print those the two uh data sets there so uh square brackets square brackets and we're got te Texas and then Texas one year that we want to look at here okay so then to plot we're going to plot this exact same data set like this and that should be good um we can try to run that I don't think that's going to work though because we're going to have a lot of not availables here yeah I me uh I wasn't expecting to see that um interesting we may not actually let's see so we did not let's see HPI data TX one year equals that resample print uh TS one say resample how oh okay that's our problem is this how open high look closed uh mean let's try that one more time there we go okay so yeah so we got the Texas monthly but the TX one year is not displaying the reason for that is basically it's filled with knotted numbers but there are some valuations here so after you get more than uh 12 of them like let's do this you do actually have values right because they are uh picked up once a year now but we have all these missing numbers right not not a number not a number not a number then finally we got a number okay so there's a lot of options here um we'll kill it kill it um so the first one that we were going to say was to delete it so we have a couple options when we go to delete data there are um first you can just drop any existence of a NADA number okay so our first choice would be just drop na so we can do something like this we could say this as na um then what we do is I can come over here and I can go well actually we'll leave that first one and then we'll print another one uh we'll just do the head though and then we'll do it again so we'll print out two heads and then what we'll say is HPI dat. dropna and that's all that'll drop the not Avail not of numbers and it will return the data frame so again remember what happens when we return data frames we have to say in place equals true or redefine the data frame one of those two options okay so we're going to say drop na what this does is any road that contains any not of numbers gets removed from our data so we save and run that sure enough cool we got the graph but then we also see that not only did we get the graph but we also um the the regular Texas is also only sampled once every year okay um and you can see that happens down here so this is the original where was every month basically and these are just not a number because there's no number for that and then here there's no more n numbers but that's because it's being sampled every single um every year so uh so let's close this out now um so that was drop Na and that would that's a scenario where you drop e every row that basically has any sort of missing data now the next option you have is you can say drop na a but let's say you only want to drop rows that are have all not a numbers okay so so what you would do here is you would say uh how equals all in place true okay and um so that's that's when every not a number right so we don't even get a graph now because not every here was not a number only one column so it's only when everything in that data set is not a number that it would be dropped so there you go uh there is one more parameter it's called threshold and the threshold you can set the minimum um so pandas actually pandas drop na let's look that up really quick move this over so this is from the documentation and you can see these are all your options you could just ignore that how any all default is any um and then threshold here this is uh the basically require that many non notavailable values so if there are that many non notavailable values it won't be removed okay and then in playay is equals true so moving this over here um so that would be how you can remove by just drop all noted numbers and you can kind of have you have a lot of choices you can drop if there's any if it's all not a number or if it passes a certain threshold of not a numbers uh we drop those now another uh option here is to fill not a number so instead of droping a let me just highlight this we'll come down here paste um instead of droing a and we'll leave in place equals true for now uh our option could be Phill Na and then with fill na you have a method and your method is one of two things you might well there's more but they basically boil down to the same one uh you've got forward fill or backward fill so you could do F fill that means forward fill you can think of this kind of like a broom because it kind of goes um I don't know it's a little counterintuitive to me at least initially when I first learned it but forward fill that means you're going to take data and fill it Forward okay uh so that means it's actually going to take data from before right right so to me when I hear I would hear back fill and I would think oh okay it fills from the previous data maybe I don't know but anyway think of it like a sweep you know or like a broom you're kind of sweeping the data forward so this will take previous values and fill them in forward so we can run that real quick and sure enough we get a graph and there you go this is it and you get all these little steps because it's like it's taking all the this data filling forward boom new data awesome fill forward boom new data fill forward okay now you have all also you've got fill backwards so that was forward fill but then you could do back fill and that would be taking data from really after and filling it backwards so this one would be a little more biased right and sure enough it it kind of precedes this line in every aspect because it's taking data from before or it's taking data from the future and filling it backwards and that's also why it fits this line apparently much better so we'll close this out okay so that's fill in a and then before that drop na okay but we also have another version of fill Na and that is um we can fill it with a value so we can say what we want this value to be now in this scenario it makes no sense but the reason why you might fill na is you could fill it with really anything you want you could fill it with literally the characters null uh you could also uh like with machine learning for example a lot of times people will take um not a missing data and fill it with like negative 99,999 something like that and the Machine learning classifier will automatically see that as a significant outlier and kind of treat it like junk basically and kind of ignore it uh so you might see something like that where uh instead of method B fill you can fill na a with the value uh of and then we'll just do 99999 and then and in place equals true that's fine let's [Music] do save and run that and then so you get a graph like this um obviously that's not it doesn't really work out for us cuz we're doing 9999 and so it's an ugly graph but uh in the data you can see here where they were not in numbers is now filled with 900 or 99,999 now um an inter in one here is let me move this over uh with fillna a you have this choice of limit uh so this can help you kind of um keep a little bit of data Integrity here so if you have let's say you know a data set that has 100,000 rows but 50,000 of those rows is going to be 99,999 uh you that might not be uh good enough for you that might be a little too many that you filled in so what you can do you can take this limit and apply that limit and say I will only do this to say you know uh 100 rows or you know you can make it Dynamic and and check the length of the data frame and say okay if the if and set the limit to be 50% of your data frame or 10% of the length of your data frame and what happens um is you can if you apply this limit after you apply that limit um let's see p is contains na I think it's um is na is what you can ask for uh and you can find out if the data frame has any noted numbers within it so uh and if you do what you would do is you would run through uh your data frame you would apply maybe your replace you would set your limit and then you after that after you run that fillna a check to see if there's any NADA numbers in existence in your data frame uh and then if there are either you know stop doing calculations or uh whatever now what I'm trying to find is check see in pandas data frame trying to look that up because I'm pretty like I've done that before let's click check this yeah so what you can do is um to check to see just for the record Google is your friend uh what you can say here is you could say print uh DF do is null do values. suum and let's say value fill that and then we could say limit equals 10 okay and let's run that real quick wait for root oh DF my bad let's do that DF doesn't exist for us okay so we come over here and sure enough we can see that we've got 414 remaining noted numbers here so that's our way of checking so like after we we do this fillna a and we say hey we only want you know 10% of our data frame to contain Ned numbers you could run this and if that this greater than zero uh you know break or something like that stop doing stuff uh send me an email and tell me something's wrong um anyway I think that's good enough for handling of missing data in the next tutorial uh I think what we'll do is we'll be talking about rolling statistics and stuff like that so you can do some pretty cool stuff there uh and then after that we'll talk about handling erroneous data so anyways that's what we're going to be uh doing in the next tutorial so stay tuned for that questions comments suggestions whatever leave them below otherwise as always thanks for watching thanks for all the support and subscriptions until next time