S&P 500 Web Scraping with Python

what is going on guys and welcome to this finest tutorial in today's video we're going to talk about web scraping now one of the first videos on my channel here was the stock visualization tutorial using candlestick charts and what we did there was basically using the Panda stage reader in order to get some financial data from the Yahoo Finance API so we passed the ticker like a APL or FB or TSLA for the respective company and then we got some data process this data restructured it and then in the end we visualized it with MPL finance but one thing that we couldn't do with that and we can still cannot do with that is getting some sp500 data or actually getting the data of any index so we cannot get a Dow Jones Industrial Index data you cannot get a Nasdaq index data you can also not get data for foreign indices because Yahoo Finance API this is the way we did it DF equals web data Rita Apple Yahoo start and so we specify the ticker we specified the API we specify the start and we specified the end date now what we cannot do here is we cannot just visualize the S&P 500 a and now we can maybe somehow find ETFs that represent it but you cannot get the companies listed on the S&P 500 you cannot pass S&P 500 as a ticker symbol but you may have reasons to want to do that so maybe you want to know which companies are listed on the S&P 500 maybe you want to calculate the S&P 500 index and of course you want to do this in your script so you don't want to just look it up for yourself but you want to you want to have this data in your script so what you have to do is you cannot use the app and Finance API but you have to find another source and of course you can look for some CSV files or something but those have to be up to date and it's hard to look for them so what we can do here is we can use web scraping to get them from pages like Wikipedia so what we can do here is we can go ahead and type list of S&P 500 companies and here we have a list of the companies I don't know if this list is up to date but I think it should be kind of update so here we have all the ticker symbols that we can use then for accessing each of those companies individually via or via the Yahoo Finance API so in this video we're going to learn how to do that we're going to use web scraping in order to get this data into our Python script so let us get right into it now before we get into coding what we need to do is we need to figure out the structure of this website so what web scraping does is it takes the HTML code of a website and extracts the information that we need so we need to know how this website is structured and what the HTML code looks like so what we do is we right-click somewhere onto the table here and click on inspect element or anything similar in other browser so I'm using Firefox here and then you can use this tool here to basically click on what you're interested in and we're interested in this little box here or in these little boxes here and what we can see here is that we have a table basically an HTML table here which is the whole thing basically and then in this table we have a hat and a body and in this body we have table rows and all these table rows are essentially just obviously the rows of the table and in each table row we have table data which is basically just a column off this row so we are interested in the table data the first table data element of each row because that's the symbol we're not interested in the other columns here we're only interested in the ticker symbol because that is what we need for a Yahoo Finance API or for other services so this would be basically neat we need to extract the table with the class wiki table sortable we need to take the first table off this Patriot that has this name because of course down here we have we'd scroll a little bit here but down here we have another table which we're not interested in so we are just going to look for the first table and from this table we're going to take all the table rows and of those were going to take the first table data elements so the first cell basically this what we're going to do and now we'll get into the Python code so for this script we're going to need the beautifulsoup library so it will going to do is we're going to run CMD and we're going to activate the Cana environment if you have one and then use pip to install beautiful soup for this is the web scraping library that we're going to use it has a little bit its name is a little bit odd but we imported SBS for not as beautiful soup for and then we give it an alias off ES and what we also need to do is we need to import your request module which is part of the core Python stack so we don't need to install anything for that and request module is what we're going to use in order to get the HTML data from the website so what we're going to do is we're going to send an HTTP request to the website we're going to get the HTML code as a response and this HTML code is then going to be fed into a soup object so we're going to use beautifulsoup to create a so-called soup object and this soup object is going to scrape through the HTML code to and it offers us a lot of methods to do that in order to filter out the data that we're interested in so first thing is we're going to say HTML equals our requests dot get and now we need to pass the link here the sp500 link so we copy it and enter it here as a string we pass it as a string and then we can also go ahead and say print HTML text because HTML itself is just a response object if you want to see the HTML code you need to say dot txt and here you can see we now have the full HTML code in our in our list here in our scripture sorry so actually if you wanted to you could use string functions to filter out the data yourself but we can use a soup object to optimize this process to make it a lot easier so we're going to say soup equals BS beautiful soup and we're going to pass HTML dot text so we now have a soup object which is based on the HTML data that we provided and now we're going to define what we're looking for so first of all we want to have the first table so we're going to see first table or just table equals soup dot find and we want to find a table object or table element with the attribute and now we're going to pass a dictionary for that the attribute class shall be equal to wiki table sortable now depending on the time you're watching us this might have changed I don't think so because this works for two years now at least two years now but you can check out the Wikipedia source code or eight actually just the HTML code to figure out if this is still a case however this fine function here gives us only the first table elements so we're not using find all or find multiple of those we're just getting the first occurrence of this table or off a table with the class wiki table sortable so this is the first table because our table is the table we're interested in it's the first table on this website here or at least the first wiki table sortable I don't know maybe Wikipedia structures its page so that's some of these elements here are tables I don't know but this is just a table and now we want to do is we want to get all the rows every single row of this particular table and then from this row we want to get the first column the first cell so what we say is we say first of all we need to define a ticker list where we're going to to store all the tickers so we're going to say tickers equals empty list and we're going to say rows equals table and now we're going to use find all not just find find all we want to find all the elements that our table rows so TR with one exception because if you look at the website here if you look at the table here we're not interested in this head here so we're not interested in symbol security and so on we're interested only in all of those rows here so not not the first one so basically we're just excluding it by saying skip the first index we're starting at index one up until the end so we're using index slicing here to get all the rows so this returns a collection or a soup collection you could say of all the row elements and we of course can iterate over it so we can say for every row in Rose what we want to do is we want to say ticker the ticker that were seeing in this iteration is just the row find all the table data off this row and pick the first one and from this one the text the HTML text then we say tickers dot a pent ticker so again we're doing is we're getting the first table all the rows iterating over the rows and for each row we get all the table data pick the first one which is index 0 and get the text of it which will be the ticker symbol and we append this ticker symbol to the tickers list so this is actually it we can now go ahead and print the tickers and you'll see that we we will have all the sp500 tickers in the list here so you can see the only problem here is that we have a backslash and so what we can do is we can actually not a pentacle ticker but skip or remove the last two characters so basically up until minus 1 should do it cutting off the last two characters as you can see it works so we have all these ticker symbols here and now what we can do is of course we can export them to a CSV file we can use them directly to iterate over it so I'm not going to complete all the the whole kalsec project again but let's say we have a data reader now what we would do we would say for each ticker in tickers we could say I know what was it web equals not equals DF equals something like data readers or web data reader was it I think and we could say ticker and then Yahoo and so on and then once this is done we could take this data frame and save it so you could save the whole S&P 500 companies and all this data into a CSV file so you don't have to access it you can then also of course compute the index or basically calculate the index value however now you have it and if you don't want to get it download it every time you run it what you have to do is you have to just say with open for this we're going to need tickle though so we're going to see you pickle with open I don't know s SNP I'm going to call it as NP so I don't have to use the + sign SMP 500 dot pickle as file or SF and then we're going to say pickle dot dump and we're going to dump the file into this file we're going to dump the tickers and now we should get a new file here no we didn't get a new file no such file oh sorry need to open it and write binary now it works and of course we can now I think at least we should be able to do this we can now just say import pickle and then we can say with open eyes & p500 pickle read binary as f umm it was something like pickle load F yes this is data or actually you could call it tickers and then we can save print tickers I think this should work there you go it works as you can see this is how you get all the data from Wikipedia how you web scrape it save it and then load it again into your script so that's it for today's video I hope you enjoyed it and I hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below also if you're interested in more finance or web scraping tutorials also let me know in the comments section because then I can react to it and make more videos to these topics and if you haven't done it yet subscribe to this channel to see more future videos for free often that thank you very much for watching and see you next video bye [Music]