Matplotlib Tutorial 8 - getting data from the internet

Key Takeaways

hello everybody and welcome to part eight of our data visualization with python with matplot lib tutorial Series in this video we're going to be talking about how we get data from the internet so uh what we're going to go ahead and do first is we'll get rid of this this kind of commented out code you might you can save that for your own records if you want but we're going to start building just with numpy so first of all we we want to like you know pull some information offline so how do we actually go about that well first of of all you know sources of data online are very uh vast you've got places like quando and I'm pulling up all the places uh that we'll talk about here you got quandle and um and then all kinds of apis like Google has apis and Yahoo has apis and even like Wikipedia has apis for stuff but we're going to focus on um well we're not going to use quondal actually in this in this series probably but this a really useful place to gather information but we're going to use Yahoo finance's API so um that's just for the record but like you can look up the quotes here and then you you can find the link to the quotes but we'll end up just writing it in and I'll give you a straight link but we'll be using Yahoo finance's API to get stock prices so what we're going to do first is uh let's start by first of all to get that data we'll get rid of this plotting stuff and we'll do uh let's let's create a function out of this so we're going to Define graphcore data and we're going to graph data for a specific stock so this is kind of the same sort of code we'll end up building something pretty similar to what I've built in the past uh where you can just type in the Stock's name and it'll graph that so for now we we'll push this over and then we'll have down here we'll call graph data and then we'll pass through a stock for now let's do uh let's do Tesla so TSLA that's for the electric cars so uh graph Data stock and then so what we're going to start with is um first we need the stock price URL stock price URL is going to be um this will be HTTP colon slash and I think I will just have us I'll put the link in the description it's pretty certain in I'll forget to put the link in the description so if I forget someone do please remind me and I will put the link in the description as soon as possible but here is the link I don't want to waste time typing it out but basically this is the finance API uh this is where we input the Dynamic stock whatever that happens to be and then we have chart data and this is the other this is one variable and this is another variable we're just hard coding it but this is like 10 years of data that we want one thing to note if you decide to go up on your own own little tangent here and you decide to change this to things like one day when you're using anything greater I think than 10 days uh the prices you get or the date stamps are in like real like legible date stamps uh or at least somewhat legible but if you get down below 10 days you're going to get Unix time we'll cover how to handle Unix time but just understand if you try to get ahead of me you're going to be like H unconverted data um and that's why so anyways uh moving right along stock price URL good got it next um we're going to say the source code of that URL is going to be equal to and we are going to have to incorporate URL lib for this one so let's bring in um actually we're going to need a few things on top of this so let's go ahead and import URL lib and then we're also going to do uh import Matt plot li.es as MDES so map plot lib has its own little date converter called MDES or date handle called M dates so we're going to utilize that anyway source code is equal to URL li. request.url open and then we want to open this stock price URL and then we want to read and anytime you read stuff in Python 3 you'll you can just add the decode there um we're going to end up decoding it anyways this may not actually be useful but we'll put that there anyway uh then then we're going to say stock data is going to be an empty list and then we're going to say the split Source equals the source code. spit by a new line so the first thing that we could go ahead and do is we can actually visit this URL to like see what we're working with here so let's take that copy paste that and I'm going to replace that you know quote stock with Tesla so that pulls up this okay so this is the return that we get now Tesla hasn't been around for 10 years so we don't get 10 years of data but we get like basically 5 years really really close 5 years actually anyway uh so here's our data and basically we can see there's some useless data up here but then as once we get down here this is all just stock prices okay so that's what really what we're after and here this is the year 2010 this is the month 06 this is the day the 29th and then you've got open well you've got close Yahoo finance orders is funky but you've got close the high of that day the low of that day the open of that day and then the volume of that day so volume is how many shares of that company were traded so our our source code is going to look basically just like this now we're splitting that Source Code by new line then what we need to do is we kind of want to filter out all this junk okay so the the way that I've done this in the past is to first of all check to see if the length of that line split by comma so splitting by a comma you should have one element 2 3 4 5 six okay you should have six elements when you split by comma but and then the only other line that has six elements when split by comma is this line right this has there's one element 2 3 four five and six so we also just want to just hardcode the filtering out of that line and other than that we should have all pricing data so we write a quick um for Loop to basically handle for that so what we're going to say is uh for whoops not in all caps for line in Split Source what do we want to do well we're going to say split line equals each or actually we just say line split line equals line dos spit and we split by comma and then what we're going to do is if Len of the split line so this is where we're asking if it's equal to six it's got to be exactly six if it's less than six we don't want it if it's more than six we don't want it so great if that's the case great and then we also just want to get rid of that line that has like values date close high low I just use use values so if the term values is not in the line what do we want to do well we're going to say stock data. upend uh and we append that line so now we have the line of data that's basically the year stamp like this we've got basically this data right and this will be you know a long line and then there will be that's the list element then here's a second element the you know third and onward so now we've got that the next thing that we want to do is we need to kind of use numpy to read this data into variables so this data goes as date close price and I usually do close p uh because close has a a meaning and so does open for example so we have close uh High P low P open p and then we have volume and we say that we're going to unpack to those as we showed before is equal to np. loadtxt and the text that we're loading the text is just this variable right so you can load all kinds of stuff with load text it's not just text uh we load stock data I'm going to start inserting commas or I mean of course I'm going to have commas I'm going to start inserting uh new lines here so we don't run off the screen but stock data uh then our delimiter is equal to a comma then unpack needs to be equal to true and then finally we add a new thing and that's converters and we're going to say converters is equal to and you use a dictionary for this and you say zero so that we want to convert the zeroth element that would be date because date right now is just this weird format okay so uh M plot lib is just not going to recognize that so we convert that and then we're going to say in Python 2 this is relatively simple we use MDES and stuff and we can convert this and it works uh but in uh Python 3 it still for some reason does not want to be decoded right so we have to write our own quick little function that's okay so the converters is going to be we're going to convert the zero using the function of byes um P dat to num and then we specify what the structure of this date is basically and so there's all kinds of symbols in date time conversion uh I'm just going to show a few here but usually at the end of the day you use the following okay you've got percent uh percent Y is like a full year so 205 for example and then you've got percent lowercase y this would be a partial year uh so you might have 15 right then you've got um so that would be year percent m equals the number month uh percent D would be the number day and then you've got uh percent H which is equal to hours percent uh hm would be minutes and then you've got percent S capital S and that is seconds okay so just keep keep that in mind that there's those you can look up more if you want but these are the ones that we're going to basically need we actually aren't going to be even using hours minute seconds here so in here the structure of this is a percent capital Y followed immediately by a percent uh lowercase M follow follow immediately by a percent lowercase D so that would be percent y percent D percent uh or actually percent M sorry and then percent D if someone had a date that was like let's say the date was um 126 2014 let's say so month day year you would have this would be structured as percent M then a dash then percent d a Dash and a percent capital Y okay so when people add dashes in there you you can add them in as well so anyways um now we've got the converters now we just need to make that conversion uh function so we'll go ahead and um I think we'll store the conversion function for the next tutorial and then we'll work on actually graphing this data as well in the next tutorial we should be able to get this information pulled and graphed within one more tutorial so anyways that's what we got for now if you have any questions or comments up to this point uh please do feel free to leave them below otherwise as always thanks for watching thanks for all the support and subscriptions and until next time