Web Scraping with Professional Proxy Servers in Python

Key Takeaways

[Music] what is going on guys welcome back in today's video we're going to learn how to do professional web scraping via proxy servers in python so let's get right into it alright so what is the idea behind web scraping via a proxy server instead of just web scraping without a proxy server the basic idea is that when you do large scale professional web scraping the problem is that your ip is probably going to be blocked sooner or later by someone so if you go to some website to web script some stock data or some book data or some shopping data or some housing data whatever you're web scraping if you do that over and over again in certain intervals and if you do that on a large scale your ip address is probably going to be blocked by the respective provider and that's a problem with proxy servers what you can do is you can change the ip address you can change the location so you go via a proxy server you don't send the request directly but you send a request to the proxy server and the proxy server then sends a request to your target so that you always have a different ip address so you can use a proxy server that has multiple different ip addresses and so on and in today's video the goal is to show you how to do that on this uh web scraping website so i'm not going to use an actual website because web scraping is annoying sometimes it's forbidden sometimes it's illegal however i'm going to use this sample pagebooks.2scrape.com here it's totally legal it's totally fine you can do that as well and here we have basically just a couple of books with prices and you can add them to a basket you can look at some detailed information this is the sample page that we're going to use today and i'm going to use that to show you how to use proxy servers for web scraping now this gets us to the next point a proxy server is fine but most proxy servers that you find out there if you just type free proxy server list or something like that they're going to be unreliable they're going to be down they're going to be super slow they're not professional proxy servers and of course if you're just doing something for 10 minutes on the site you don't need a proxy server especially you don't need a professional proxy server but if you're doing this professionally on a large scale if you actually want to do solid web scraping you need a professional proxy server provider which brings us to the sponsor of today's video however this is not a sponsorship that you can just skip because it's part of the video so it's not a sponsorship where i just tell you about a sponsor and then we move on with the video the sponsor is actually part of the video because we're going to use proxy servers from oxy labs so this is here the oxy labs dashboard as you can see and this is the aux labs website uh you can you can go to the website you will find a link in the description down below to see what they offer and we're going to use the web scraper api so the oxy labs web scraper api offers us the possibility to use those professional um proxy servers this professional aux labs environment to send requests to oxlabs and then oxlab sends the request from certain locations we can even specify the country uh to the target destination so what i'm going to show you today is how to use the oxy labs uh scraper api you can sign up there you can take a look at uh what they're offering you will again find a link in the description down below and this is what we're going to do today we're going to go to this book site here as a sample of course this is not the best use case in real life you won't scrape just books but whatever you do real life stock data whatever you want to scrape um frequently and on a large scale you want to have a professional proxy server and today we're going to look at how to do that with oxylabs all right so let's get started with the code the first thing we want to do is we want to install two external python libraries that we're going to need for sending requests and doing the actual web scraping work those two libraries are requests and beautiful soup four so request is what we're going to use to send requests and to get responses and beautiful soup for is what we're going to use to create so-called soup objects based on the html code and to then filter out certain elements so we're going to open up a command line and we type pip install requests like that then you press enter and then pip install beautiful soup 4 like that and then we can actually go ahead and just import requests and we can go ahead and import um actually from bs4 import beautiful soup like that and now let's talk a little bit about the basic idea of what wave script web scraping is for those of you who are completely new to that the basic idea is you have some url like for example this books2scrape.com and what we do is we send a request so we do something like requests.get and then uh books2scrape.com with all the http stuff and so on and then what we get as a response here is the response itself so we we can say response equals request some get and then what we do is we create a soup object so we say beautiful soup of response dot text and then with that soup object we can save this in a soup object we can do stuff like um find certain things or find all of a certain category we're going to look at that in a second here but that's the basic idea we send a request we get a response we turn that into a soup object and then we can get certain elements and do certain things here and as an introduction example here what i would like to do is i would like to go to that page here pick a category like philosophy for example and take all the books that we find here get all the prices automatically so i just entered this url into my python script here and i get all these prices in a list and then we can take something like the average price or something we can calculate the average price that's the first introduction example that i would like to start with i'm going to first do it without a proxy and then we're going to do it with a proxy so that you see how it's done with the proxy uh it's not too difficult but it's a little bit different because we have this intermediary so we're going to start simply by saying the base url of our page is gonna be what is it gonna be it's gonna be this here so we're gonna go to that there you go this is gonna be the base url and based on that based of that base url we're going to create some more complex urls like we're going to start with base url plus and in our case now we have catalog category books philosophy and whatever there you go with the index.html and our goal is now to get the content of that page and to web scrape certain elements in our case the price tag and then we want to calculate the mean price of that particular page so how do we do that we say response equals requests dot get url and then what we do is we say um soup equals beautiful soup response dot txt and i think we need to say parser equals lxml uh not txt but text there you go so to see if that works we can just print the soup object and if we have some content in there we do have some content in here this worked so now what we can do very simple is we can go ahead and say find underscore all and then i can say p so this will give me all the paragraphs um as you can see here all these p tags and i can do the same thing with ace to get all the anchors so to get all the uh what do we have here guess that parser warning no parser whatever we got all the a tags here um [Music] and this works as you can see now our goal is to get the prices so what we do is we go to the website here we right click on a price we go to inspect this works on firefox and on chrome and here we can see here that this is a p so a paragraph and it has a class price color and it seems like all these when when we look at it we will see that all these have a price color class so all we need to do is we need to get all the p's where we have price color as a class so we can say price tags equals soup dot find all and then p and i think now we need to pass a dictionary i'm not sure but i think so price color and let's see if that works otherwise i will have to double check with the prepared code but there you go you can see here now what does it say all the time that i don't don't have the parser is that correct don't i have do i not have the parser what did i do before i mean it doesn't produce any problems yet lxml yeah i don't know we can also just or maybe it's not the parser maybe i don't have to pass parser equals but just lxml i think that was the issue i always mess up these notations as you can see this works now so we have these uh paragraphs we have these classes price color and then we have the content of those so all we need to do now is we need to get those price tags and from those price tags we basically filter out um the price how can we do that we can just say the prices themselves are um price dot text for price in price tags now the problem is that here we're going to now get and i don't print this anymore here we're now going to get a bunch of problems because we don't have um we have a couple of symbols that we don't need but you can see here that it's always the same we have these two symbols so we can just cut them off now one thing you need to keep in mind here if you're using a proxy they might not be able to read certain bytes they might give you some some slightly different html so in our case you're going to see that when we do it without a proxy we're going to have to cut up a cut of the first two symbols if we do it with the proxy we're going to cut off the first symbol so just uh you need to be aware of that so we're going to say price text from position 2 and then we should get only the prices and the last thing we need to do here is we need to say float actually not here but here so float off whatever the price text is and then we get a list of floats and then if we want to get the mean all we need to do is we need to import statistics and we can print statistics.mean of prices so that is now without a proxy if you just do that you don't need a proxy because that's not something that you do all the time on a mass on a massive scale you do that once twice something like that but still we're going to look at the example with a proxy to see what you would have to do in order to run some more complicated scripts so we have to start simple and then we can build on top of that and for that what i'm going to do now is i'm going to get rid of all this here so i'm going to get rid of actually this we can keep but i'm going to get rid of that because what we're now going to do is we're going to send the request not to the website we're going to send a request to oxylabs so we're going to start by defining a function so that we don't have to write this huge chunk of code all the time we're going to call this proxy underscore request and if you're confused you can always go to the oxylabs documentation to look it up or you can go back to this video if you forget it in the future so we have this proxy request and what we do is we define a certain payload and the payload is going to be uh it's going to have a source parameter this is going to be set to universal it's going to have a url this is the target url so this payload is what we're going to send to the server the proxy server and the proxy server needs to know what it itself has to send uh or where it itself has to send something to so we pass this url parameter here and we say this is the url that we defined up here so this is the page that we actually want to visit because we don't want to visit the proxy we use the proxy to visit this page so this is the target uh and then optionally you don't have to do that you can also specify a geo location if you want to be from germany for example i'm from austria so now i'm going to be from germany i can also type uk or something and then i'm going to be from a different um from a different country and this can be also additionally important because sometimes you get different prices different results from different countries and maybe you have a website that sells books and you want to know okay from all the countries where can i get can i get this book the cheapest and then you just web scrape for every country every book or something and this is a large-scale web scraping so for that it would make sense to use a proxy and in fact you can only do that with a proxy uh properly because then you can also change the location right so this is the payload and then what we're going to do is we're going to say the response is request dot um request so this is how it's done in the um in in the documentation they do request and then they specify post as a method here and the next thing is the url of the of the oxylapse uh server so what we do is we type realtime.oxalabs dot io slash v1 slash quarries like that and what we need here is we need an authentication now you need to have an aux labs account for that obviously since this is the proxy server that you that we're using here and for that you need credentials this is a username and a password that you have for your api if you sign up if you create one if you if you get the access you're going to get those credentials i have those in a file now so i'm going to say username password is equal to open credits dot txt in reading mode dot read dot split lines and then i'm gonna just say here the authentication is tuple of username and password like that and the last thing is we want to say the json is equal to the payload that we just created above so this is going to send this package to aux labs to the respective api and now what we need to do is we need to say okay the response and this is a bit difficult because this is different than when you just get a normal response because what you get from oxylabs is you get a response json object where you have the result in a separate key value pair so you don't just get only the result you get a bunch of additional information so you need to get the um the result itself so what you do is you say response dot json and then you access the results key value pair you go 0 so index 0 and then content this is how you get the html file uh the html so you don't need dot text you don't need anything this is what you need and then what you do is you say soup or actually we can just return this immediately return beautiful soup response html lxml so now this is how you send a proxy request so instead of doing now just uh the the request to to the website itself what we're going to do is we're going to say proxyrequest url and we're going to save this as the soup object the rest stays the same so the web scraping itself stays the same the only difference you're going to see right now in a second here when i run this provided that everything works no we have a problem content doesn't work yeah because it's a it's a lowercase c not an uppercase c but you're going to see that the values are not quite right because we have cut off the first digit so in this case due to the byte encoding we need to go with um cutting off the first one because one of the bytes that we had before is not here so this is now the exact same result and we can use this now we can use this function to do all the requests now one thing that you're going to notice is that sometimes a proxy server might be a bit slower but you don't use a proxy server to speed up your web scraping and you don't do it because your web scraping is too fast you do it because you have a certain use case that you want to do large-scale web scraping you want to scrape a lot of data and you want to do it frequently and you don't want to get blocked and you want to use different locations in order to do that you need to go through an intermediary and this intermediary sending a request to this instance in this instance sending the request to the server and that server sending it back to the instance in this instance sending it to you this just takes more time obviously even if the proxy server is super fast it takes more time this is the trade-off but for that you get the stability and the reliability of professional proxy servers so let's get to the next example here let's say that we now want to go for multiple pages let's say we have a category um which one could have more pages let's go mystery mystery has here page one and two or i think we should have i think in the examples that i prepared i had some some more uh fiction i think fiction has multiple pages yeah fiction has four pages so fiction what we can do is now we can scrape all these prizes then click on next scrape all these prices click on next all these prices click on next until we don't have any any books to scrape anymore so how do we do that professionally using web scraping now first of all we would have to find this next button here so we click on it inspect it and you can see that it has this um anchor tag href tag and it's inside a list element now you can you can go by that you can look for the list element next and or you can just go for an anchor tag that has the text next because it always has the same text so this is what we're going to do we're going to now get all the books from a page then look for that next button go to that link do it again until we find no next button that's the basic idea to go through all the pages so how can we do that what we're going to do is we're going to take this category url first um so we're going to say this is the category and we're gonna get rid of the page one html because the page one html is just right now right so when we go to a category like fiction first of all it starts at index so it started it starts at index and then only when we click next we see page two page four uh page three and so on so we're going to start with that and we're gonna say then that the [Music] url in the beginning is going to be base url plus cat for category plus index.html we can also go with page one because that works as well but this is how we're going to do it and then we're going to say price is going to be an empty list and then the typical while or actually let's do it in a different way let's say next link equals true we're going to assume that there is a next link and until we see something else until there is no next link in fact uh we're going to go through a loop so while we think that there is a next link what we're going to do is we're going to say soup equals proxy request to the respective url so we're going to start with this one here which is the fiction category and then what we're going to do is we're going to get all the page prices and this is going to be the same um that we did before here so the price tags is going to be whatever we find in the soup object and uh we can actually also copy this here so we have all the prices here in a list and actually this is the prices so we're going to say page prices because those are the total prices we have this empty list here and those are just the prices for that particular page so we're going to do it like that and then we're going to say okay for um for each price for each page price that we have in page prices we're going to append that price to prices and i think there should be a better way uh can we not do something like actually prices plus equals page prices shouldn't that work as well so we basically just take all of those and we add them to prices i think that should work as well we don't need a loop for that um and then what we do is we we look for that next link so we say link equals and then we say soup dot find or actually suit body find and then we look for this a tag where the text is equal to next and if we find it so we say if link if there is something we're going to say that the next url is going to be base url plus cat plus and then then whatever this this link is i mean we need to see first of all if we have a relative path so if i go to the next button here is this a relative path yeah it's just page three so we need to do it like that um otherwise we're just gonna say next link is uh next link is false i'm gonna break out of the loop and yeah we don't need to print the url here so this should actually work and in the end what we want to do is we want to print all the prices so we want to say print prices and we want to print also the statistical mean of the prices let's see that works or if we messed up something loading first one works still and now it does something hopefully now we don't see progress because we're not printing the links but hopefully it works we will only know when when it's done there you go so now we have all these prices here from all the different pages and this is the average price so this is how you can also find buttons and we're not actually clicking them this is not a web driver this is not a browser that we're interacting with via code but this is actually just going finding an atac going to the ahref doing the same thing there not too complicated so another thing that we can do here and i don't want to do too many examples here because the the principle is always the same we use this proxy request we send this package to through oxy labs and then we get that another thing that i want to do here is i want to just do something more um specific so we want to go to a category like i don't know psychology where psychology do we have psychology here yeah let's go to psychology now and what i want to do here is i want to click on a book and then when when i'm on a book maybe that's not the appropriate book for this video let's go with this one here um what i want to know is how many of those are available so i want to get this three available here i want to get this three here and then i want to get a list of which book is available and which not so how can we do that essentially what we do is we say soup equals and then again proxy request we need to know the website now i'm just gonna okay let's do it properly i just wanted to copy paste but we can do it properly as well let's just say base url plus that there you go and what we're now going to do is we're going to say okay proxy request i'm going to send the request to the url and then what we want to do is we want to find each of those objects so the whole thing so we're basically looking for this whole container here which is an article with a class product pot so we're going to say that the books that we're looking for are equal to soup dot find all we're looking for article where the class is what was it product pot was that yeah so those are the books and then we're going to say books equals and we want to get certain things here so we want to get actually the only thing we want to get is we want to get that link that gets us to the actual book page so that we can scrape some information there so we want to have the anchor um that gets us from here to the page and for that we're going to say book dot find anchor so a um and want to get from that the href so the link and maybe we should not store this in books but in book links so book finds a href and then for book in books so this is going to give us the articles and here we're gonna um essentially get the a tag we're gonna get the link from that a tag and we're gonna get uh all the all the book links now to see if that works already we're gonna print book links but before that i'm gonna comment out all of this here so we don't have to deal with that all the time so let's see if that works or if there is a problem yeah you can see it works with a relative path so this means going back and it seems to works somewhat so what we're going to do next is we're going to say the book names we want to have those as well because the book names uh we want to print that this in this book is available uh such and such many time or so and so many times so one half is we want to have this um what is it the h3 here so we want to have um the title of the book and for that we're going to just take the text inside of the heading so we're going to say book find h3 dot text um for book in books like that and now all we need to do is we need to iterate over the book link list and we need to say um that we want to go to that page find the availability and then print it with the book name so actually since we're going to iterate over the book links and the book names we're going to do it with an index so for i in range length book links they should have the same link length so it doesn't really matter and uh now what are we going to do let's go to one of those books and let's right click here this is essentially the p so the paragraph with the class in stock availability so we're going to um just say combined or actually we don't need to call this combined let's call it url it's going to be base url plus booklinks i so will this give us the proper url i'm not sure actually because this href here is being clicked from that page and it is relative to that page so actually we cannot just call it from the base url we cannot just append it to that so we need to actually copy this as well i think so plus that like this so we have the base url which is twoscrape bookstoscrape.com this and then from that path without the index because that's the actual site but from that path here we then go back a couple of times to the respective books so we say soup equals url actually proxyrequest url and then we want to again what was the class in stock availability the piece within stock availability so the availability is going to be soup find all piece where the class is in stock availability and then we should print that to see what this is dot text to see what we get here as a result if this works or not we don't have okay what did i do wrong here oh this is a result set there's only one i should use a find function because we don't have multiple of those and now it should work right yeah so we have this we can see here a lot of space how can we get rid of that space easily by calling the strip function and that gets rid of all the things and now we also want to get rid of is one two three four five six seven eight nine 10 characters so 10 and then we have uh yeah well this is not dynamic now you can also try to split on on spaces but i think we're just going to do 13 here for demonstration purposes and then we're going to strip this thing again and maybe now first of all what's that oh maybe we should also make this an f-string and use the book name here so book names i and then this availability here that should then work i think and i hope now what is this your ayah like that that should be it right i hope so there you go there you go we get the 14 12 11 okay here we don't have the strip properly so we probably should do something like 12 here for this particular use case but all in all you can see how this goes the tutorial here is now not focusing on perfect web scraping the main idea here is to use a proxy for web scraping because when you do something like that on a large scale let's say you have 20 websites that you want to scrape all the books off and you want to do it multiple times a day and you want to find the best offers from different locations you need to have professional proxy service because you're not going to be able to do large-scale web scraping um consistently without such a professional proxy server and oxy labs again they're the sponsor of today's video they do a great job you can also see here the analytics you can see okay how many times do you use what um the average response time whatever and it's really a nice service it's really a nice um product to use if you are into large large scale web scraping they also have different products you might want to take a look you support this channel by doing so as well so check out the link in the description and this is how you do web scraping via proxy servers so that's it for today's video i hope you enjoyed it and i hope you learned something if so let me know by hitting the like button leaving a comment in the comment section down below as i already said i would be happy if you check out the link in the description down below that leads to oxy labs they're the sponsor of today's video they support this channel you support this channel if you click on the link and if you are interested in the sponsors i think it can be interesting to some of you guys who are into web scraping so make sure you check out the link in description down below and other than that thank you much for watching see you next video and bye [Music] you