Pandas Basics - p.2 Data Analysis with Python and Pandas Tutorial

Key Takeaways

what is going on everybody Welcome to the second data analysis with python and Panda tutorial video in this video we're going to be talking about is uh going over the panda Basics so you've seen your first Panda data frame and what we're doing with it but now we're actually going to code it ourselves and go through with it so if you didn't copy and paste the code from before uh basically what we're going to have is you can copy and paste that code and then we'll just start with an empty slate here uh and we don't need IOD either okay so we're going to use basically these four lines here pandas as PD again PD is just kind of shorthand it's a standard if you're going to share your code or ask somebody to help you with your code you probably should use standards uh matplot li. pyplot is PLT standard matplot lib import style uh that's so we can quickly make our graph look a little more appealing than the default so uh the first thing that we're going to go ahead and do is like let's say um so a panda's data frame if you're coming from Python and data frame just isn't sitting right with you for whatever reason uh a data frame is a lot like a python dictionary in its heart and the way that we can reference it is like a python dictionary at its heart so uh first of all let's say you're you're operating a website now eventually we we'll get into housing data and stuff like this but for now we'll just use a really basic example let's say you've got a website and you got some web was thinking stats and I typed data and then I either one was fine anyway now uh so let's say you've got some information here let's say the first bit of information is the day uh so day colon and then we'll have a list here this will be day 1 2 3 4 5 and six and then after that maybe you've got uh visitors and also again if you're not familiar with dictionaries I have the basics tutorial on dictionaries um relatively easy to understand if um if you just go visit that tutorial real quick visitors uh we're not a very popular website so we don't really have that many visitors uh see 4 34 I think that's six right yep six then finally uh bounce rate so this is the rate of how many people will come to a landing page and leave immediately not something you want 65 72 62 64 and 54 and that's not six so one more 66 awesome okay so that's actually probably industry average so 65% of the people that come to your website leave immediately shocking I know now uh the next thing is to convert this to a data frame now it's not always going to be this simple but with a dictionary it is because a dictionary just suits a data frame really easily so we can say DF equals PD that's referencing pandas now we're going to say pandas data frame note the Caps capital D capital F there and then web stats boom we have created ourselves a data frame now when you're working with pandas and you do changes or you make something a data frame whatever you know just writing it out I'm confident you know that made a data frame but I can't see it and that's why people love goys uh and that's why people like Excel over programming because with Excel they can visually see it well with with pandas you can visually see it too so what you can do is you can say print and you could print the entire data frame so DF and again DF uh is just used as shorthand um it's a standard generally if you make a data frame you call it DF but that's not that's like the least restrictive of all of these uh standards if if you want to call your data frame something else like before we called it I think XOM it was it was more it at least it made sense to have that variable name so especially if you're going to have like five data frames it would it would be very unclear even if you called them df1 2 3 4 5 that would be really messy it's it's better to call them something that relates to the data frame okay so anyway uh so you could call this you know stats or something like that but we'll just call it data frame so you can print out the data frame F5 to run it save good um and there we have uh our information you'll notice that we've got this 0 1 2 3 4 5 over here again that's the index now if you don't specify an index everything is treated as a normal column and uh an index will just be kind of generated automatically for you but later on we'll be talking about u in this tutorial how to set a specific index uh and we'll talk about why you would do that in a moment now that's our entire data frame so that's six rows now you can reference just a specific um section by doing print DF do head okay and DF head just prints the first five rows 0 1 2 3 four conversely we can also do DF do tail and that prints the last five finally what we could do is uh you can do maybe you just want to print the last two well you can throw a two in there as a parameter and this will print only the last two columns so that would be right here pretty cool um so you'll find yourself using dfad basically every time you know you apply maybe a calculation you create a new column based on some logic or something like that it's always a good idea to just kind of output that that data frame to the console so you can actually see what you're doing if you're working in U most interactive development um editors you're going to see the data frame anyways every time you redefine it so um so that's one of the bonuses to using like an IPython notebook a lot of people have been asking me hey man you ever heard of IPython notebook yes I have I just don't like them very much I'm trying I want to like them very badly I just simply don't anyway uh so that's uh printing out the actual data frame so you can see what you're doing now let's set an index why might we have an index first of all the index is basically what how all of your data is either related or how you want to visualize it related so for us maybe we want to relate the day uh like you know the most obvious index for this data would be day right so if you have any sort of Time series data generally the index is going to be the day uh but if you've got a website maybe you would want your index to be visitors and then you want to know you know maybe what are the you know you've got visitors but then you've also got bounce rates and maybe you've got you know referrals and stuff like this and you want to see how how everything interacts with a certain data point like visitor account or something like that you know you never know what you might want to do and you can have multiple indexes uh I wouldn't I've never needed multiple indexes but you can do it and there's a few reasons why you might do it but it doesn't make much reason to cover that right now so the way that you can set an index is is just DF do setor index to uh and we'll say day now if I do something like this print DF do set index today and run that you'll see the return is the entire data frame with um day now being the index but then later on we're going to you know be epic programmers we're going to do a bunch of calculations blah blah blah blah blah and then we go print dfad and then we're going to we see this first one which is the way we wanted it and then we get this second one and it's we're like what what happened here we lost our Index right that's really annoying and uh this is something that's going to slap you in the face a few times at least but what happens here is when you modify the data frame like this what what actually is happening here is when you say DF do set index day you are returned a new data frame okay so that's why this print statement works worked out for you you're you're returned a new data frame and that's going to be the same thing if you're in interactive development uh like an let's say IPython notebook you're going to be returning the data frame it's going to look great and you're going to be like yeah cool and then you're going to keep going on and then suddenly your your index is no longer well what's happening is it's just returning a new data frame so what you have to do is one of two things one one way that you can handle for this is you can create uh well I'm going to leave this code here just simply because I usually post up theod code but actually well I'm just going to comment it out um anyway one way that you can do it is you can say something like this DF equals DF do set index today and then you could print you know dfad and that's going to work just fine okay but that's kind of sloppy and but you could also say like maybe you want to make a new data frame out of it so you could say df2 okay so they leave that there so you can make your choice if you want to set a new data frame so obviously df2 equals that so then if we print a df2 head you would get the newer right so they give you the option whether you want to keep your original data frame or not now what if you don't want to do this like double you know this equals nonsense well the other way that you can do this is you can just say DF do set index today and then you can specify in place equals true and this will modify the data frame right then and there and we can say ds. head run that and uh there you go so that happens so uh you can try to internalize this as much as possible I still get slapped by this maybe you guys are better than me but uh that's something that's going to hit you if you find out that you know these various changes you're making just for whatever reason aren't showing up it's not just when you set the the index there's a bunch of these uh commands that you're going to do and then you're going to think it worked and then later on you're going to realize it didn't work cuz you didn't either set it in place or redefine it moving on uh let's make some some space here uh to reference a specific column so right now we only have two columns you've got day as the index and then you've got the column of visitors in the column of Bounce rate to reference a specific column it's pretty simple you would say DF and then like it's a dictionary square brackets and then visitors visitors we can to save and run that and there we go we got just the column visitors just the information and all that now you might actually want this to be like a list or even ire array we'll talk about that don't you worry close this uh another way that you can reference just a specific column is you can use uh as if it's like an attribute so you could say print DF do visitors okay so we'll get two outputs of visitors here identical there's no difference okay so if you'll notice up here in bounce rate I put an underscore this allows you to use that dot uh like it's an attribute uh if you want want if you leave a space that's totally fine you can reference it like this but you obviously cannot do do bounce space rate that would be a syntax error right so you can't do that okay but of course you could do you know bounce rate um actually let's just return this uh you could do oops you could take bounce rate with the space here copy paste save and run that and that's totally fine you can do that so you might find data sets have spaces there so you won't be able to do this kind of notation but you can you can still handle spaces that's fine so now the next thing that we're going to go ahead and cover is so that's how you can reference like a single column if you wanted now how do you reference multiple columns if you wanted you can reference the entire data frame as we've already shown but how do you reference multiple columns well what you can do is you could say print DF like you're going to reference a column only this time we're referencing a list of columns columns so two you know square brackets within square brackets and this is a list okay so we can say we want to reference and I'm going to go ahead and add that undercase again because I want that uh there now we can reference bounce rate visitors visitors okay so that'll print um just those columns now obviously we had other information and so those are the only two columns that we had but if you just wanted to reference two columns that's you would do it finally how do we reference how do we convert this to maybe a list okay so what we could do is we could say print DF do visitors visitors. two list I'm going to comment these things out right here save and run that and sure enough that's been converted to a list now with python there is no such thing natively as an array you have lists and then you've got multi-dimensional lists like lists within list list list in a list is totally valid but as you will find here in just one moment well what if we did this what if we took this original DF that so we'll say print uh that information. two list so right those were the two columns of data what if we tried that ah shuck we can't do that not two list that's not going to work out so it has no attribute because it's a it's it's being treated like it's an array so what we're going to have to do instead is uh come to the very top import numpy as NP cuz that's what the people do and then we'll do this instead of2 list print np. array so we're converting this to an array close off everything save and run that bada boom bada bing you have your list up here and your array down here now of course you could convert this to an array too that's totally fine uh but for those of you who are familiar with other data science libraries numpy arrays are basically integral okay so uh that's what you have to um do to convert now the final question would be of course you know can you do this can you say df2 equals pd. data frame and we want to convert that print ef2 save and run it wouldn't you know it there it is so uh let's say you're working I don't know with um open CV and you've got things that have been converted to pixel arrays can you put it into a panda data frame you sure can so really easy input output obviously here we've just done dictionaries and umpire arrays but we'll be talking about how you can do you know CSV files and stuff like that uh as we continue along the series so that was a lot of information being thrown at you in a relatively short amount of time we're coming up on 15 minutes so I guess not too short but it was a lot of information if you have any questions or comments concerns suggestions whatever feel free to leave them below if this was too fast or whatever again there is a text based version of this tutorial on pythonprogramming.net I will usually link to that specific tutorial in the description otherwise this is a data analysis tutorial you go to pythonprogramming.net dashboard data analysis there it'll be so anyways uh that's where you can go if you're looking for more information um otherwise as always always thanks for watching thanks for all the support and subscriptions and until next time