Dynamic Javascript Scraping - Web scraping with Beautiful Soup 4 p.4

Key Takeaways

what is going on everybody and welcome to part four of our web scraping with beautiful soup 4 tutorial Series in this tutorial what we're going to be talking about is how to scrape dynamically updated information from a web page so to begin I have added some information to the parks me mcface page underneath this picture you can see this JavaScript Dynamic data test and it just says look at your shining it says that because we're viewing it with a client in a a browser and the browser is actually doing something that makes that show up let's look further so viewing the source code zooming in scrolling down here is what we're looking for so this is what we I was just showing you whoops okay so what's happening is this paragraph text is of class JavaScript test it's it's got an ID of yes or no or yes no JS and the starting text is why you bad though but we didn't see that that's just the starting text underneath it we've got a script so JavaScript and what it's doing is it's finding the element by an ID that ID is yes no. JS which is here and when it finds that it's going to say theiner HTML so we're going to say okay now the HTML says look at you shining that's what we want it to say so when Google Chrome or whatever browser you using browses this page it says look at you shining it requested data from the server and the initial information that the server said was no no no between the paragraph tags this is what you want to say but then it also said here's a script though by the way and then Chrome ran that script so actually you run that script the server didn't run that script you ran that script so closing this out if we wanted to parse those tags we would use the following code you don't need to write this out I mean you can if you want we're going to use it in a moment well towards the end but hopefully if you've been following along you should already have everything up to this just without these two lines anyway this is basically the code that we would use we're going to parse paragraph tags with the class JS test simple enough so let's save and run that real quick and we get why you bad though that's not even what we thought we were supposed to get so a lot of times maybe you're going to parse a web page that maybe you're going to parse a table you're expecting that table to have some data and lo and behold it has nothing like what the hell so the problem is you aren't a client you're not a browser so what we have to do is mimic being a client or a browser and actually run that JavaScript which is actually a little bit more involved than you might think um or maybe you are thinking that and realizing oh no so so there's a whole lot of options add or disposal for how we're going to do this I think the easiest way to do this is to use Pi QT uh specifically we'll be using qt4 I'm sure you can do it in qt5 but I'm just going to use qt4 so I do have a tutorial on Pi qt4 you don't need to follow this entire tutorial but you should go to the first step the first page and you'll need to get qt4 if you're on Windows go to this URL here and download the wheel for pyqt4 there might be an installer now or something really simple you can check that out if you're on Windows you can go to Riverbank Computing and I think you can install it from there it's been so long I don't remember what method I used on Windows but I have it it's possible if you're having problems uh leave a comment below and I'll do my best to help you out so once you have qt4 you're ready to rumble so we're going to go ahead and come up above these Imports and we're going to start importing a few new things first we're going to import CIS we're doing that because QT applications want to be able to take arguments and it's going to be angry if you can't so we're going to do it then from PI qt4 and I'm going to go ahead and just copy this so I don't typo it but it's capital P lowercase y capital Q lowercase T4 dot QT we're going to import Q application then again and in fact actually I might as well do copy all of that from PI qt4 do QT core we're going to import Q URL by the way I didn't really explain it um Q application is probably easy enough but it's the thing for making applications ql this is how we're actually going to read the URL basically and then finally um QT webkit actually it's capital K webkit import Q web page lovely so this is going to let us actually load the page page and act like a browser act like a client and run that JavaScript so it's saving us a ton of programming that would um be involved there and in theory you could actually make the page show up even you can make your own web browsers in qt4 as you might be able to surmise at this point anyway um cool so we have all the Imports we need and now what we what we need to do is we're going to write a client class and if you're not too up to speed on object-oriented programming hopefully this one will be super simple enough I don't yet but very soon I should have an object-oriented programming Series so if you did if you are watching this sometime after the original upload date you should check out that if you need help anyway class client client is going to inherit from the Q web page um and we're just going to make some slight modifications just tiny ones uh Define and we're going to do our init method it's going to take self and it's going to take a URL of some kind what we're going to say is self. apppp is equal to this Q application it accepts cis.org oh my gosh cis.org V got it then we're going to initialize the Q web page Q web page. init self then self. uh load finished. connect self. on page load so we're initializing the Q web page or initializing the we're starting up the we're defining the application in the we're starting it up it does this will have its own in it as well but anyways we are starting up the Q web page then what we're saying is we're connecting this method here when the load is finished what do we want to do so initially what we could do without defining this client we could actually just work directly with the Q web page um with a Q web page object access the URL and then use this Mainframe do2 HTML stuff that I'm going to show you really quickly you could do that but the problem is the HTML you would get would be nothing it would be blank it would have nothing there uh because the page hadn't actually loaded because Pi qt4 is an asynchronous Library so it runs asynchronously and it's not going to hang up so it's going to keep doing code for you uh so when you go to say okay this is what the HTML is equal as soon as it just initiates that code it's then going to give you the HTML we don't want to do that but instead we want to do is when the load is finished we want to run onpage load whatever onpage load might be we need to Define onpage load so Define on page load obligatory self uh self. app. quit that's all basically we just want this to run uh until the page loads and then when the page loads we're done easy enough now what we're going to say is let's see client let's do just move this down for now I'm going to pull some of this information so to start we'll say the URL is equal to this cool then what we're going to say is the client response is equal to client and then we're going to pass that URL which will go through the init method then we're going to say the source now the source code rather than being the URL lib request we're going to let um QT handle the source and all this which is kind of what we're we're doing up here we're at least letting QT load the page and then now we're going to grab the source code from uh Q web page basically so the source is going to be the client response because this client response is a client object client object is inherited from the Q web page so we can use the Q web page methods now so client response. Main main frame the main frame dot uh oh shoot is it I think yeah two Capital HTML so that's going to say okay this this basically Q web page object let's get the main frame you know the the frame as in the the thing that we're looking at and then convert it to HTML you could do um I don't know if we'd actually be able to do it I don't want to do it but you you probably could do a DOT show uh here we'll do it after if this works we'll do a DOT show just to see if it works anyway um I'm pretty sure it's do show Source although we're running the app it's probably it wouldn't work I don't think we we could show here I don't know anyway I don't think it's going to work uh so the source we've handled otherwise everything else stays the same so we don't need to do anything else uh let's save and run that uh print jst t. text none type has no attribute text what did we do wrong I'm blaming you all let's see client self that have to quit oh we didn't finish so so okay so stupid okay so um after we say basically we needed to tell it what to do when the loading is finished before we actually load anything so what should have really given it away was the fact that we have URL here and we never actually do anything with URL great okay so self. main frame and then we use load and we load the Q Q URL and then the Q URL loads the URL and then um the app will uh exit because we don't actually care that much okay let's try it one more time boom we got what we wanted all right so that is um that is how you can scrape Dynamic data Just for kicks I want to see if I can actually show I don't I don't think it's going to work but yeah yeah I guess that's probably more I don't know I'm not going to mess with it anymore anyways uh so that's how you can scrape Dynamic data no one came here so they could see it show so I'm not going to waste any time trying to make that page show but just know that Pi qt4 really is uh it's basically a browser and you can make a browser really easily with qt4 just a little bit more code uh and we we would be able to make it a browser so um a few things to Note One basically you know it's running it's running it's running it's running it's R oh there we finally got a response that took a while okay why did that take a while well we would have to look a little deeper but there's basically whenever like with let's say web scraping especially there's two pain points one pain point is probably um loading Pi qt4 loading up this web kit stuff loading up our browser client thing um that that's going to take some time to like set all that stuff up okay and then we're also going to process whatever JavaScript is there we have to process that JavaScript that's going to take time okay so that's one thing also when you're parsing websites the other thing is just latency and response time of the server so a lot of people have asked me as I was releasing this series uh what do we do about the fact that beautiful soup is slow beautiful soup is not really that slow I mean it's it's a fairly uh efficient framework the problem is a a server request and response time is probably going to be like 500 milliseconds or more okay and so it's not instant so if you're trying to crawl 500 URLs that 500 milliseconds is suddenly 250 seconds for all 500 URLs that's a really long time to wait so the two things you need to think about is if say you're you're mimicking a client for that uh you're going to need to utilize multi processing and just like I don't have yet a tutorial on objectoriented programming I will have a tutorial on multiprocessing probably within a couple weeks so uh again if you're watching this not immediately or if you are watching this immediately uh stay tuned for that uh so if for at least for using QT and stuff like that you should use multiprocessing so if that's on your side now if it's a latency issue and you're just simply waiting for that response that means your CPU is actually idle you've got idle threads thus you can use the threading module and that will speed up the whole process and in reality you're going to probably need to use both okay you're going to want to make full use because you're going to be waiting on other people's processing sometimes and then many times it's just going to be a bottleneck of your own processing so uh keep those things in mind if you're having speed issues otherwise uh that should work for you guys so if you have questions comments concerns whatever feel free to leave them below otherwise as always thanks for watching thanks for all the support subscriptions and until next time