pandas best practices (1/10): Introducing the dataset

Key Takeaways

hi there this is kevin from data' school you're about to watch a panda stew toriel that I presented at a Python conference in 2018 the goal of the tutorial is to help you become more fluent at using pandas to answer data science questions I split the tutorial into ten parts this first video introduces the tutorial and the data set and the other nine videos contain exercises that we work through now I recommend that you watch the videos in order and if you want to do the exercises at home you can get the data set from github and there's a link to that in the description below this is an intermediate tutorial so if you're brand new to pandas I recommend that you start with my other video series data analysis with pandas and there's also a link to that below I hope you enjoy the series so let's get started alright welcome everyone thank you so much for being here it's real pleasure my name is Kevin Markham and this is using pandas for better and worse data science it's my privilege to be here today so before I start talking about the tutorial and what's going to happen I'm just gonna tell you real quick about setup so that if anyone's who's not done setup yet can go ahead and do that and so what you need to do if you haven't already is go to the github repository I've got these bit ly links bit dot Lea using pandas and even if you've already set up your system go ahead and go to that github repository and once you're there you can get the data files if you haven't already there's some instructions on the page for how to download them if you're not familiar with downloading from github and then you'll want to open your Python environment you can use any Python environment I recommend using the environment that you're most comfortable with I'm going to be up here using the Jupiter notebook but you do not need to do that at all just use whatever your you're comfortable with when you've got those files downloaded put them in a place that's easy to access usually it'll be in the same directory as your script or your notebook and that'll make it easiest to access the files I've got some instructions on the page for how to check that pandas and matplotlib are properly installed those are what we're going to be using in the tutorial and if you're having internet problems or you are just like wait I forgot to bring a laptop with pandas I have I have five flash drives up here all five of them have the data files on them if you need them and can't download them and then three of them also have Linux Mac and Windows versions of the Anaconda distribution which is pretty big but you could definitely install it in a few minutes if you need it so if anyone needs one of the figures out they need one of those flash drives just come up here and I'll give you one okay all right so while people are getting settled and set up I'll tell you a tiny bit about myself and then talk about the tutorial as for me you still living in Washington DC and recently moved to Asheville North Carolina in September with my wife and my son I'm the founder of data school and I teach data science in Python online I'm mostly known for video tutorials on YouTube anyone ever seen one of my videos cool cool thanks for watching and I used to do a lot of classroom instruction in data science through general assembly in DC really enjoyed that so it's nice to be back in front of a classroom because right now I teach online only and I was I was at PyCon 2016 doing a tutorial calls it call it called using what was it called it was machine learning with text in Python two years ago and that was really fun and that's not so let me talk a little bit about the tutorial format so you know what to expect the goal here is to help you become more fluent at Panda is to solve data science problems okay so I am assuming you know some pandas this is not an introductory tutorial I don't assume you know much or anything about kind of the data science process we're gonna be doing a lot of exercises this is very exercise driven and were between the two data sets we're gonna start with the police data set and then at some point we'll move on to the other data set depending upon timing how it'll work is I'll give you some sort of question or task and you will try to work to answer that question with pandas and then you know I'll give you a couple minutes or however long seems reasonable we'll talk about the answer together and each exercise will conclude with you some takeaways for here the things I was trying to teach you through this particular exercise and some of the lessons will be very like pandas specific lessons and other will be more general data science lessons in terms of the pacing we are please keep in mind that everyone in the room is going to be at a different experience level and some of you who are more beginner might find that this goes way too fast and others who are more advanced might find that it goes too slow and I'm gonna kind of aim for the middle so I'll be kind of watching how people are doing during the exercises and answering questions and at some point I will just kind of stop and we'll move on so I'll do my best to kind of split the difference and head for an intermediate level if you do finish early please help your neighbor please help each other I'll be walking around but there's only one of me so feel free to help each other and if you've if you're at any point struggling there are more seats throughout so feel free to wander wherever if you find that you're struggling and you're like boy this is really hard don't worry about it it's not a big deal there's gonna be most of the people in the room will not get everything and it's okay if you're only picking up certain pieces totally fine and the other cool thing is that most of the exercises are completely independent so if you miss exercise 1 3 & 4 & 6 & 7 you're gonna be fine on 8 9 and Beyond so don't worry if you miss something you will be fine there's a couple times where that there's that's not the case and I will say you need to type this line of code so that you can it will work for future exercises so you are not gonna get lost or I hope you won't get lost we will help all right final thoughts before we start if you have not used the Jupiter notebook before which as I said is what I'm gonna be using it's probably not the best time to learn right now because it's a very different kind of environment then you're you might be used to you can ask questions at any time and if it's beyond the scope of what I want to talk about today then I'll just tell you and we can talk about it afterwards if I even know the answer to your question the break as she said the break is at 3 o'clock it'll be hopefully from 3 o'clock sharp to 3:20 and then we'll start right back up we end at 4:40 so it's 3 hours of actual content and after the tutorial I will post all of the code on github so if you want to follow along and type everything I type you can if not there'll be reasonably well commented code on github afterwards that you can refer to all right enough me talking on and on any questions for now Before we jump in any questions awesome awesome ok so I have I've got my Jupiter notebook open then I'm going to be using and we're gonna start just doing some imports and then we're going to talk about the first data set briefly and kind of do a little refresher on pandas before we actually do the first quote exercise okay so of course we'll go ahead and import pandas is PD and if you don't use matplotlib much no worries but the standard import is import mat plot plot lib pie plot as p LT okay how is the font size in the very back is it okay bigger alright alright how's that alright and if you are in the notebook but not if you're not in the notebook this is irrelevant but if you're using the notebook for certain versions of the notebook you need to say % matplotlib in line ok if you're not in the notebook that's not relevant and it's no longer relevant for certain versions of the notebook but it'll allow plots to appear okay so first data set is set one is from the stanford open policing project okay and i can tell you the URL it's linked this will only take a minute we're just going to go to the web page of the data set and you can scroll down on the github repository to this link to stanford open policing project let me make that bigger and this is not critical if your internet is flaky it's that Wi-Fi is open it's 2018 pike on this is the stanford open policing projects website you don't need to read it just saying this is a data set of 31 states and police traffic stops in those states okay and if you did come to this page i want you to click that view data button and then it'll show you if you're interested you can see all the states and what data fields they have on the bottom but really I wanted to bring you to this there's a link here that says read me and not critical but it's a readme file that talks about some of the columns at times in the exercise it might be helpful to have this page so if you're trying to decode something in the data set you can always come back to this page and you can ask me later if you if you got lost how to get here and I can pull it up on your machine but this is some data about the data set the data set we're using is from Rhode Island and I just picked it because it's a nice small data set but a lot of the fields are there I decrease the size from the one that's available online by randomly sampling it 20% of the rows and removing some of the columns that didn't interest us so that's those are the only modifications I've made everything else is as is okay all right so let's go ahead and read it in to pandas and then again we're just going to talk about it briefly and then we'll jump into the first exercise so I'm gonna I'm going to create a data frame named 2ri short for rhode island because it's nice and short to type but you can use something else if you if you like and this will actually just be a simple read CSV command of police dot CSV okay and you shouldn't get any errors on that and you don't need to change any of the default read CSV parameters alright so starting now I'm going to be asking lots of questions and throughout the tutorial so lots of participation so this is the head of the the data frame what does it look like each row represents and either you can raise your hand or just call it out an incident a traffic stop so you know it looks like okay a date a time and age so this must be each row is one traffic stop okay and then if you use RI dot shape the shape attribute what are these two numbers represent I'm sure someone said a number of rows number of columns so ninety one thousand seven hundred and forty-one rows fifteen columns and then let's just take a look real quick of the D types and I know we we barely look that the data set you'll have plenty of time to do that during the exercises I'll focus you on certain columns but here are the data types so I just need a volunteer to explain what these different data types mean real quick who wants to do that okay object is a string so probably some sort of category could it can't actually be an arbitrary Python object but it's usually a string and then float and bool so what are those right so the float floating-point numbers and bool short for boolean which are true/false values okay so just useful to keep in mind what the different column types are that we're going to be working with and let's scroll back up for a second back to the head and we've got this na n here in the county name column what does na n mean not a number and more generally like what does it indicate missing value someone said or lots of people said so na n indicates a missing value it's a special value in pandas that indicates the value is missing so why might a value be missing why might it be missing it wasn't recorded at the time that's one that's the usual answer great and can anyone tell me one other reason a value might be missing one more time data corruption okay great great great any other reasons yep in the back redacted for privacy at great answers the one other I would bring up is that it's not like a value can be missing if it's irrelevant for a particular row so if you had a data set of Python users and there was a column called like attending Pike on 2018 that was true or false and you had another column called Pike on 2018 registration date that would be a missing value for anyone who's not registered for Pike on so it's not that we didn't want to collect it it's that it's irrelevant okay so because the person is not attending Pike on if they don't have a registration date so you would make that like a missing value so the point there is values can be missing for different reasons sometimes you wanted to get it and you couldn't sometimes you have it you got rid of it and then sometimes it's just not a relevant value okay and why would we mark something as na n rather than just like a zero or an empty string or the text unknown what's the value of having na n how about there right right you want to distinguish he said you don't want to screw up like your statistics on a column so basically you want to be able to distinguish what is real data from what is like marking like if you typed the word unknown in a string column and that was like a last name field what if someone's last name was unknown or how do you distinguish the missing data from the real data so that's why you mark data as missing okay all right I swear we're almost at the first exercise I'm gonna just type one more line of code and I want you to tell me what we're doing alright dot is null dot sum okay all right all right that isn't all that sum I have two questions what do these numbers mean and then second is going to be how does this line of code work so what are these numbers the sums of all null values in each column or more generally just like what does this represent at a high level the count of missing data in each of these columns okay so stop date has zero missing values stop time has zero missing values etc now my other question is how does this line of code work who feels like they can explain like the two steps that go into this line of code okay right so our first step is our I dot isn't all which is a data frame method and it returns a data frame of true values and false values and then when you take the sum okay you're taking a sum of a data frame and you might think well what's that going to do well it does two things one is it sums over the zero axis the index axis so it kind of collapses all the values down like the the math I think of it as the mathematical operation goes down or you can think of it as the dimension of the data that's being collapsed is the zero axis so it does the sum in that direction such that you end up with column sums and the other thing it's doing is R it is translating true into one and false into zero and if you didn't know that true actually is equal to one and false is equal to zero which is why you can add together boolean values okay all right so um let's sorry I let me delete that to clear the screen here okay so let's get into our first exercise and get you guys and girls typing and writing some code