Pandas with Python 2.7 Part 9 - Statistical Information

Key Takeaways

what is going on everybody Welcome to the ninth pandas and python Basics tutorial video in this video we're going to be talking about uh some statistics Gathering things so the last thing we did was um standard deviation but there's also a whole slew of other things that we can do uh so I'm just going to show you a few more things as far as Gathering statistics about the data that you're looking at so with that let's go ahead and get started so uh let's do I guess we can keep High minus low but the rest of this stuff let's just go ahead and delete and we'll make some space that's good enough so the first thing that we can do is we can use uh do describe so we can say DF do describe like that and this is going to give us um some basic information like rows the average of all the rows and actually we need to print this out rows average of all the rows uh the minimum the maximum and some other stuff so anyway uh we come over here [Music] and this gives us data on all the columns basically so here we have the amount of rows so everybody has the same amount of rows okay so it's 2830 rows um this gives us the average of each row so the average of opens is 1172 highs and so on um the average volumes the average adjusted close blah blah blah the standard deviation for the whole set the minimum for the whole set um 25 50 75 and then the maximum for the whole set so those are like stages in between so the minimum the you know 75 or the 25th percentage number you know and all that kind of stuff so as far as you know with high minus or the high or the I'm trying to think how to describe this I'm having a very hard time if you couldn't if you couldn't tell what's the minimum number what is the you know uh bottom fourth number so on uh anyway enough on that um so just some basic numbers um on the on the whole data set so we'll close out of this uh the next thing that we can do is correlation so basically this is going to give us the correlation data so remember the like we plotted like a scatter chart and we want we kind of used a scatter chart to decide for ourselves if there was any sort of correlation between volume and then you know High minus low and all that kind of stuff so volume and volatility pretty much um so but we can actually find the correlation using a pretty simple query or command uh do core so we can go ahead and save and run that and that gives us correlation of pretty much everything here so this is correlation of open to open um you know so it's a one right it's perfect one as opposed to a perfect I don't think we have any perfect negative ones we don't even really have anything that's really not correlated well anyway it gives you the correlation so the correlation of say high to open they ought to be pretty close right so when one changes significantly probably the other one did too so that makes a whole lot of sense what about volume volume to high minus low right so here's volume and then High minus low so volume to high minus low point4 correlation so I mean it's somewhat correlated it's not zero but it's not a perfect one either but at least not a negative one so we can sort of deduce that a lot of times when volume is high the high amount is slow is also a large number um and yeah so anyway those are just some correlation statistics on all of the data in the data set so it's really quick you know if you wanted to calculate correlation yourself um or at least for me uh the processing time to do that because I would just use a for Loop and iterate through you know each one You' build it I mean I it would take a lot longer than the instantaneous number that we got there anyway uh the other thing we can do is covariant so CV um a lot of people have conf confusion about what is covariance and what is correlation um myself included I would say that at least my understanding of the difference is covariance is almost like a measure of the strength of the correlation like correlation is kind of like a looser thing you know they always say like correlation is not causation right whereas covariance is more like the strength of that correlation um so maybe it's a one step closer to causation but anyway uh a very confusing topic if you try to figure it out and if you ever look it up online like people are pretty much interchange the two whenever you're trying to find a really concrete answer what is covariance you'll find like an identical definition to what is correlation right so anyway um the way I understand it is like Co correlation is just kind of like a loose thing and then covariance is a little bit stronger and then covariance is like a strength of correlation but someone feel free to correct me if you want um anyway so there is core variance again this would take a lot longer to calculate yourself I'm pretty impressed on how fast uh pandas will calculate these two things um the other thing that we can do is we could say for example um we could do something like this so we could say print DF uh dot like if we want to just compare two right so we could say print uh DF um a list within it and we want to say what's the correlation let's say uh volume to um you know High minus the low and then do dot core um we'll go yeah save and run that and this gives us the correlation we already kind of picked this up uh before but but anyway obviously volume to volume is one and high high to low is you know. 39 or whatever now uh interestingly enough um it's surprising how fast this is and how builtin this is uh this is actually a very popular method of data analysis especially when we look at things like Forex but also Stock Investing and stuff um people make these massive correlation or Co variance tables just to find stocks that are correlated or um 4X ratios that are correlated with each other and then that way when one reacts first they invest immediately in the other one because they're they're expecting that one to react with it and this actually is a fairly decent strategy that people employ um and people pay actually a good amount of money for services that do this for them there's websites that like sell just this um so I think it's kind of comical that it's like built right into pandas and it's pretty simple to do I mean I think that anybody who's never programmed before could figure this one out pretty quick and save themselves uh paying someone else to do this but anyway for example what we could do is and nobody has to follow me exactly with this uh I think probably what I'll do is I'll just copy and paste some data into here and then I will link to the copy and paste just because otherwise it will take a really long time to do this so first off let's go ahead do we already have yeah we got dat time imported and all that so let me copy and paste this and I'll explain what uh what exactly I'm doing here yeah this will work I suppose okay so we'll copy this and we'll come over here paste and so what we have going on here is we start here so here we're pulling just a bunch of data from Yahoo finance right not really that much actually but a bunch of data uh we're pulling data for City Group Apple Microsoft Tesla Bank of America Blackberry Chipotle eBay JP Morgan Starbucks Target and Wells Fargo and then what we're doing was we're printing the head of C uh just so you can see it what what we've done this might not be the most pythonic way of doing what I'm about to do but the reason why I'm doing this is we take C strip away everything except for adjusted close um so we're deleting all the other columns then we're saying core comp um so this is a compilation of correlation equals c and then what we're doing is we're renaming The Columns so what remains was the adjusted cloes for City group so I'm just renaming that to see in plays true and then now we're saying the column of core comp for Bank of America equals Bank of America's adjusted close column and so on we're going all the way through so now what we've got is for core comp we've got a bunch of columns here that are strictly stocks uh and adjusted close for those stocks now some people want to know like what what's the point of adjusted close versus close most of the time 99% of the time it's not going to matter but what adjusted close is going to do for you is if there's like a stock split or something like that it's going to adjust for you instead of having some wild crazy number there so we use adjusted close um for that purpose now uh moving on so we're going to do all of these then we print the header of that file and then we're going to print out core comp. and that's going to be the correlation for all of these companies so chances are Bank of America uh JP Morgan and Wells Fargo are going to have a slightly you know pretty high correlation probably above s um you know Blackberry uh I don't know whose Blackberry is going to be correlated with here but I'm not really sure these other ones maybe Tesla I don't know is there techy stocks but I don't know uh anyway let's go ahead and run it and see what happens so um save continue okay it's going to take okay took a second to grab the data but we got the data so obviously City Group City Group perfectly correlated Bank of America to City Group 97 almost a perfect correlation um no surprise big Banks um interestingly enough Microsoft to City Group possibly just because they the S&P 500 together uh quite a few close correlations but then we see like Blackberry to City Group negative Blackberry to Bank of America negative Blackberry to Microsoft which is a surprising negative um I would expect that to at least have been somewhat positive um pretty much Blackberry to everyone is negative so far that's kind of interesting Blackberry to everyone is just negative um I forget what our time frame is here but probably because Blackberry has been sucking lately what we got 2011 to 2014 yeah that's probably why my's just been going downhill I think uh so that's probably why but JP Morgan is City Group again very positive very high correlation um really so correl see Target and chipotle kind of a weak correlation well as far trying to find some stuff that isn't with Blackberry Wells Fargo to Appo apple apple um you know so anyway what people do is they take massive tables like this like really they would probably take the entire Russell 3,000 or even even more and compare them while and find really kind of obscure connections um not really sure that makes the best sense to do that I think you still need to have a logical because again correlation is not causation and if you search for it you'll find plenty of stocks that are apparently correlated but not correlated but people do it anyways um especially with Forex um and sometimes Forex it's better you know if you look at it with human eyeballs and then you'd be like oh I guess that would be correlated or something but uh anyway like I said people pay a kind of a lot of money uh for this and you do it right here within Python and pandas so anyways thought I'd show you guys that real quick if I forget to put a link to this uh someone remind me uh so you can play with it and you change up the names you could use um Forex ratio symbols instead and all that so anyway uh that's going to conclude this tutorial if anybody has any questions or comments feel free to leave them below as always thanks for watching thanks for all the support and subscriptions and until next time