Getting all company pricing data in the S&P 500 - Python Programming for Finance p.6

Key Takeaways

what's going on everybody Welcome to part six of our python for finance tutorial Series in this video what we're going to be doing is working with the list of the S&P 500 companies that we just grabbed and now what we want to do is actually get the all the data on those companies so that's what we're going to be doing here so um we're going to need a couple new Imports actually so first of all we're going to grab uh import import date date time as DT we're going to import OS so daytime for the same reason we used it before OS is to create new directories for us uh then uh pandas we'll need pandas as well so import pandas as PD and then also import pandascore dat reader. data at as web and that should be all of the uh new Imports that we need now we've already saved the S&P 500 tickers there's actually no need for us to have that anymore I'm going to create some space now and we're going to begin a new function I'm going to call this function uh get data from Yahoo so let's go ahead and Define get data from from Yahoo I'm also going to have just a simple reload SP 500 equals false well this will do is basically because we're no longer going to just be calling this raw we're going to call it in this function as we continue building functions here basically what's going to happen is each fun one function is going to build on the the previous function so we'll run them to test them by just like writing them out like this but in most like in reality it's going to be one final function that's going to just work from these previous functions so anyway reload SP 500 basically this will just let us basically if reload SP 500 if someone says yes that's true I want to do that basically tickers [Music] tickers will equal this what we just wrote there now if they didn't say that else we're going to say with open and we're going to open that tickers file right here so I'm actually just going to I'm just going to copy all this paste uh we're not going to write bytes we're going to read bytes now we're going to say tickers equals pickle. load F so then we'll have our tickers list once we have that um the other thing we're going to do is we're going to take all of the stock data and we're going to we're going to convert all of them to their own CSV file and we're going to store them in their own kind of like in a directory basically the reason why we want to do this is generally when I'm working with an API or any sort of data collecting task especially if I'm like parsing a website or doing something that takes a while uh I like to take the entire data set if I'm like all the data that you're willing to give me I'm going to take it all I'm going to store it that way I don't have to reping you because as you'll soon see in a second here um getting all the data from Yahoo takes a long time it's going to probably take you 20 30 minutes or something to get to regrab all the data um and to do that every single time we want to iterate through everything and get pricing for all these companies it it just would get outrageous but if we store all of the data locally we won't have to go through that mess so um that's why I typically will do that but in in this case I know that I'm pretty much for I think for this tutorial at least for the first many videos the only column I'm interested in is adjusted close so in theory I could take all these and combine them all in one function rather than saving it all and then later combining it and saving that but there might come a time where hm what if what if I want a high minus lows low column uh do I want to spend another 30 minutes pulling from Yahoo I already spent all that time why not just save it so as long as the data is not huge which in this case it's not it's easily saved I think it's it's like less than 500 megabytes I'm pretty sure so um yeah then I want to save everything I can so anyway we're going to do that so to do that we're going to say if not OS if not os. path. exists we're going to check to see if something exists we're going to check for stock DFS so if that directory doesn't exist os. [Music] MERS um stock DFS so if that directory does not exist locally go ahead and make it now we're going to specify start and end times same as before dt. dat time uh and just 2000 January 1st I'm going to go ahead and copy this paste 20 16 1231 and that's actually end and now we're ready we've got tickers we're ready to iterate through the tickers and grab all of this ticker data so for ticker in tickers um what we're going to do is basically um H I'm trying to decide what I want to do here you'd probably want to have another one like this is reload SP 500 you'd probably want to have like another list that was like reload data or something like that um but what I'm going to go ahead and say for now like you probably want to encase what I'm about to throw in here with a relo if reload data do this else just go ahead and and rewrite all these data frames but I'll just keep it the way it is for now if not os. paath because right basically what we're trying to do right now is just like simple research so I'm not thinking that we're going to update the prices every day but these are daily prices so in theory every single day you would re-update all of this data but anyway moving along um if not os. path. exists exists and now we're going to check for that full path so this in this case it would be stock DFS um SL something. CSV so we're trying to see does this CSV file exist and then it the the formatting here will be for the ticker so if that file doesn't exist we're going to say DF equals web. data reader and we're going to read whatever for the ticker is from Yahoo and then start end then DF do2 CSV and it's this right here so I'm going to copy this paste that's the path to the CSV else print um already have whatever it is and then we'll do dot format ticker now um just in case some people are watching that who have like really really slow internet or whatever um you don't have to do the entire S&P 500 so to continue following along at least for like the first many videos in this series there's no need for you to grab all 500 companies so if you don't want to grab all 500 companies don't there's no you won't be missing out on anything super special it's just you won't have all the data um but you're not going to really miss anything so you could say I just want the first 25 tickers or 100 tickers or 250 or whatever so if you do have a slow internet connection you know I would say just grab like at least 10 or something like that just so you can sort of follow along but anyway because 500's even on my internet with pretty good internet it's it takes a while so anyway but that's for the entire ticker list whatever ticker list you might have then we can run get data from Yahoo um and I'm just going to start running it just to make sure it's actually working um and then otherwise I'm going to cut it cuz we're not going to say here for 20 minutes and watch this run um uh we probably should have said something about where we were that's okay ah I kind of want to print ticker just so I know it's working I'm pretty sure it's working I I see it doing nothing so yeah okay so now we're grabbing I can't move it it's running oh my goodness so I got multiple screens it ran onto a different screen anyway okay so it saw that it already had like a lot of companies already and then now it's just going through pulling all these companies um so just to make sure everything's working as we intended let's open up python for finance there's our new stock DFS and sure enough here are all of the CSV files um in question cool so it's just populating them again it's going to take a while so far since I've been Yik yacking we've got uh 110 already so that's not too bad but yeah it it can take a while now I also have not found that Yahoo throttles um this is like a lot of requests in a short period of time I haven't needed just because saving the data frame takes a long enough delay to the point where I haven't needed to throw in a time. sleep anywhere but you might find that if you're getting throttled by Yahoo um I.E basically we just we just did 100 if in 10 minutes we still don't have another new 100 we've probably been throttled so you might want to add like a time. sleep or something like that but so far I haven't had any problems with it so okay great we've got all the data from Yahoo uh basically in the next coming tutorials we're going to be combining that data and then searching for relationships so if you have any questions comments concerns up to this point feel free to leave them below otherwise I'll see you in the next tutorial