Ultimate Guide To Web Scraping - Node.js & Python (Puppeteer & Beautiful Soup)

Key Takeaways

hey what's going on guys so I recently had to scrape a website and get some data and it gave me the idea to do kind of an ultimate web scraping guide so what we're going to do is take a look at some slides and just talk about what web scraping is uh why you would do it we're going to look at the steps to take and then we're going to jump in and we're going to scrape a e-commerce website for some books so we're going to get a bunch of data uh on these books and we're also going to be scraping data from a site that has pagination so we'll have to basically use our tool to to go through and go through each page scrape all the data go to the next page and so on and I'm going to do this both with nodejs and python so with nodejs we're going to use a tool called Puppeteer which is a web scraping Library it allows us to use a basically have a headless browser within our code and then for python we're going to use a library called Beautiful soup and we're basically going to do the same thing so if you want you can watch both and you can kind of compare and contrast or if you're just interested in the the JavaScript you can watch that part if you're just interested in the python you can watch that part all right but ultimately this is going to give you a really good understanding as to what web scraping is why you would use it and also how to do [Music] it this web scraping guide is sponsored by bright data their scrape off solution allows companies to SC scale up their web scraping and data collection with their Global proxy Network browser automation tools and web scraping apis you can scrape data from any public website bright data's tools offer Universal compatibility with just about every programming language and web scraping tool you can run your pupper selenium and playright scripts on full hosted browsers with unlimited scalability you also get 24/7 support anytime anywhere so if you like this guide and you want to take your web scraping projects to the next level click the link in the description visit bright dat.com okay so we're going to look at some slides talk a little bit about web scraping why we do it how we do it and then we're going to jump in and I'm going to show you how we can scrape a website using both node.js with Puppeteer and Python and beautiful soup all right so what is web scraping as I mentioned web scraping is the process of extracting data from websites this can be done manually but it's often more efficient to use a program to to automate this process and web scraping is is commonly used to collect data for research for analysis and for other purposes so for example you may want to collect data on a job board or collect product information from an e-commerce website and web scraping allows you to automate that process of collecting this data so you can save time and effort now obviously apis are one of the best ways to get data from a website but not all websites have apis so web scraping can be a good alternative however that brings me to my next point which is the the legal and ethical considerations and this is something that's really important to understand when you're getting into web scraping it can kind of be a controversial topic because some websites don't allow scraping at all of their content they usually list this in their policies some do allow it it just you know fullon whatever scrape as much as you want and some have certain stipulations so it's important to be aware of the the legal and ethical considerations when you're scraping websites and to make sure you have permission to scrape those sites that you're targeting and in this guide we're going to be scraping a site called books.to scrap.tf it's been around for a long time the the very first tool that was used I guess you could put web scraping in quotes here it was created in 1993 and that was the worldwide web Wanderer which was a bot that would go from link to link and index the content of pages that it visited and although it wasn't a scraper in the traditional sense it laid the groundwork for automated web data collection and search engine crawling so it was kind of the origins of of search engines and then in 2004 the first traditional web scraper as far as I know is was beautiful soup and that's a python Library that's still used to this day and we're going to use in this video and allows developers to extract data from HTML and XML files and since then web scraping has become a common practice for collecting data from the web and they've gained even more popularity recently due to the increasing amount of data that's available on the web now web scraping has a lot of different uses so these are just some of the more common ones so first we have data collection so obviously you know collecting data from websites for research or analysis this could be data on products jobs real estate properties news articles and the list just goes on and on price monitoring so monitoring prices on e-commerce websites to track changes in prices over time you could do this just for yourself to find the best deals or for a business to track competitor pricing uh content aggregation so you can aggregate content from multiple websites to create a new website service you can build a news aggregator that collects news articles from multiple sources and display them on a single website lead generation so collecting contact information from websites for sales or marketing purposes and uh just keep in mind you know ethical and legal considerations when you're doing stuff like that and then market research so collecting data on competitors or or industry Trends from websites and then social media marketing or social media monitoring sorry to collect data from social media websites to track Trends or sentiment now I'm just going to give you a quick summary of how web scraping works and then we'll go a bit deeper into the steps and then ultimately I'll show you how to do it with code so web scraping Works usually by sending a request to a website downloading the HTML content of the page and then extracting the data that you're interested in and you would obviously do this through code and there's a few ways to extract data from a web page depending on the structure of the page and the type of data that you're looking for some common methods include using regular Expressions parsing the HTML content with a a library like beautiful soup or using a tool like Puppeteer to interact with the page in what we call a headless browser and these are powerful because we can interact with the page as if we were a user in a browser so we can do things like click buttons fill out forms and we can extract data from the page and many tools that are used for web scraping are also used for just testing in general and they allow you to automate tasks in a headless browser and you can do what you want with that data such as save it in a file or save it in a database now as far as the tools and and libraries that we can use for web scraping this is just a few of them and these are mostly just JavaScript and and python but you can do web scraping in other languages as well so first we have Puppeteer which is a node.js library that provides a high level API to control a headless Chrome or chromium over the dev tools protocol and it's a powerful tool for automating tasks in a headless browser like I said clicking buttons filling out forms things like that and of course extracting data we also o have Cheerio for JavaScript which is a fast and flexible lean implementation of core jQuery but it's designed specifically for the server for node.js so it's it's a popular tool for web scraping in node um beautiful soup is one of the the most popular libraries for web scraping in general this is a python library for parsing HTML and XML documents it provides a simple API for navigating and searching the HTML content of a page making it easy to extract data from web pages we also have Scrappy which is another python framework for web scraping and that also Prov provides an API for extracting data handling pagination following links things like that and then selenium is a popular tool for automating web browsers and it provides a web driver API that allows you to interact with a web page in a headless browser now selenium is often used for testing web applications but it can also be used for web scraping okay so now let's go through the the main Steps to Web scraping and these are kind of the the the general have to steps and then after this I'm going to go over some some other steps that you may have to apply depending on the site that you're scraping so of course the first thing you're going to do is identify the website and you want to make sure you have permission to scrape that website and scraping without permission can be illegal and as I mentioned there's legal and ethical consider considerations when it comes to web scraping so just be sure that you're aware of all the rules and regulations that surround that in your country and for that specific website so you want to inspect the page once you've identified the website you want to inspect and determine the structure of the page and the data that you want to extract you can do this by right clicking on the page and selecting inspect in your browser or use a tool like the the browser Dev tools and you want to examine the HTML structure of the page to identify the data you're interested in and this includes looking at things like class names IDs and other attributes of the elements on the page okay once you do that once you've identified the data and you've checked out the structure you then want to write your code to extract that data you want to uh it'll have to make a request to the website download the HTML content extract the data and this usually works similar to how you would select elements within JavaScript or jQuery in fact Cheerio is is actually a subset of jQuery so it's very very similar and you may have to handle things like pagination and I'll talk a little bit about that in the next slide and we will be doing that in our code as well and then once you write your code you want to obviously run it and scrape the website extract the data that you're interested in and then you can save that data to a file to you know a Json file CSV file or you could save it to a database I mean there's different things that you can do with that data now these are some additional steps and considerations that you may or may not have to deal with so first you want to handle an pagination if the data that you want to scrape is spread across multiple Pages then you'll need to handle that you can do this by following the links that are on the the next page so you know you have pagination you can click on next see the next whatever 10 or 20 listings get that data click the link get the next 20 10 or 20 however many is on that page and specify how many pages you actually want to scrape and we're going to be doing that within our scripts you want to handle any Dynamic content so if the if the website uses JavaScript to load the content dynamically you might need to use a tool like papaer to interact with the page in a headless browser um handle errors so web scraping can be error prone so you'll need to handle any errors that you know that you have in your code to ensure that your scraper run smoothly respecting robots.txt so many websites have this robots txt file that specifies which Pages can be crawled and scraped and which Pages can and these are typically used to tell search engines whether to crawl that page or not so just make sure you respect whatever links are in that our pages are in that robots file and then rate limiting So to avoid overloading the website you're scraping you should add a rate limiter to your scraper so that you're not sending tons and tons of of requests and you know overloading the we website but that's pretty much it as far as just describing the the general aspects of web scraping so now let's jump into some code we're going to start off with node.js and Puppeteer and then we'll look at Python and beautiful soup okay so now that we've talked about web scraping let's get into it we're going to start off with node.js and Puppeteer and then we'll move on to Python and beautiful sup so this is the website we're going to scrape it's books.to scrape.on as you can see it's it's meant to be scraped so no legal or ethical issues here um and that's something you always want to look into before you scrape any website for any data so it's B it's set up just like a regular e-commerce website we have a bunch of books here for the products it has an image a rating title price in stock add to basket button and I believe there's 20 on the first page and then there's 50 pages so pagination is something we're have to deal with we're going to have to go to the second page thir how however many pages we want and get the data from each page and as you can see in the URL it goes to SL catalog page- whatever the page number so that's something we have to use and take into account in our script now as far as the data that I want I just want to get basically everything here but the image I want the title the rating which is in the form of stars and then the price and if it's in stock so so we know what we want to scrape we want to get the books I'm going to save it to a a Json file and we know the fields that we want the data so next step is to look at the structure and we can do that of course by right clicking and inspect and then I'm going to click this icon here and just select the first item and you can see that every book every item here is wrapped in a li tag and in that is an article tag with a class of productor pod okay so they all have that that article with that class so that's something to take note of cuz that's what that's the class we're going to basically use to grab on to each product and in that we have an image container which I'm not interested in I don't care about the image the rating you can see it has a class of star rating with a bunch of icons but it also has a class of whatever that rating is in this case three if I were to look at this one here you can see that this this div uh or paragraph rather has a class of one because it's a one star rating okay so we get the rating and then in the H3 is a link that has a title attribute so that's how we can get the title and then for the price there's a product price div and then in that is a price color uh class on this paragraph that's where the actual price is and then there's also a a class for if it's in stock or not so that's the the structure that we need to pay attention to and I'm just going to kind of keep this open while we uh while we write our script all right so like I said we're going to start with node.js so if you have no interest in node and puppet tier you can skip you know move on to the python part but I'm going to open up my terminal and first thing I'm going to do is just create a new directory for this I'm going to call it web- scraper Das node and then CD into web scraper node and then I'm just going to open up vs code or open up whatever text editor IDE that you want to use so from here I'll open up my integrated terminal and since we're using node we're going to npm and knit a package.json I'm going to use the Dy flag to skip any questions and I'm I'm assuming that you guys have basic JavaScript knowledge and a little bit of nodejs knowledge and for puppeteer we can install that with npm so let's do that obviously you need to have node installed on your computer to use npm okay so that will set that up package.json you can see now we have Puppeteer installed all right so one other thing I want to do I want to use the es module syntax the import syntax so to do that in node we need to go to our package.json and we need to add in here type and I'm going to set that to module if you want to use common JS and the require syntax then you don't need to do this but I prefer to use import so now I'm going to going to create a new file I'm going to call it scrapee.net you can use an iffy if you don't want to use a named function okay and then underneath that I want to make sure that I call it before I forget and then I'm just going to kind of go step by step now Puppeteer has what's called a headless browser so you can think of it as like Chrome or Firefox or whatever without the chromium without the the UI you basically do it from within your code so we want to launch that browser and I'm going to put the object that we get into a variable called browser so we're going to set that to Puppeteer and then there's a method called launch all right now I want to basically create a new page to work in so we're going to say page and set that to I'm sorry this needs to be await because these are all asynchronous so then we want to await on that browser object we have uh a method called new page okay so now we have this page to work with and we need to go to a specific URL so let's put the URL in a variable so cons URL and we're going to set that to https and it's going to be books. two scrap.tf and I'll just put back ticks here and just say page title and we'll put in the title and then the last thing we need to do after everything is close the browser so we want to say await on browser. close so now let's run this so I'm going to come down here and run node scrape because I called the file scrape.on and and books to scrape sandbox so we know that we're we're able to make the requests we're able to get the website we create this this page and we get the title now I don't care about the title I just wanted to use that as an example so let's get rid of this and Puppeteer gives us a method called evaluate that allows us to run JavaScript in the context of the page and we can then select data using just like query selector and query selector all so what we're going to do is create a variable called books and I want to set that to await page do evaluate and that evaluate is going to take in a function just going to pass in an arrow function here and we want to we want to get the the element that surrounds the data that we want to select and again if we go back to the structure here we have an article that has the class of product _ pod So within that is all the things we want so we'll use that as our as our selector all right so let's go back into evaluate and then from here we'll go ahead and create a variable called book elements and we're going to set that to document Dot and then we're going to use Query selector all because obviously there's there's more than one book Element and the selector we're going to use is the class of of that article so product underscore what was it pod okay so that will give us our elements now I'd like to show you what that gives us but we don't want we can't console log within the page. evaluate because you can think of this as like a like a a virtual page um but what we can do is return from this book elements and then outside of the evaluate we can then console log books okay because that's we have books that's set to the evaluate from the evaluate we're returning book elements and then we're logging books so let's save that and then let's run this node scrape and what we get back is what's called a node list which is a an array likee structure and you'll see that we have 0 through 19 that's because there's 20 pages on the first sorry 20 books on the first page and we're only dealing with the first page which is this URL now they reason that it's giving us an empty object is because we haven't specified any data that we want to extract yet so it's just giving us an empty object so what I want to do is instead of returning book elements directly let's convert it to an array by using uh array. from and in that we're going to pass in our book elements and then let's use map so we're going to Loop through each book okay so we'll say for each book then I just want to to return book and then we're going to run this again node scrape and now we get an actual array of empty objects because I still haven't selected any I haven't extracted any data yet so the first piece of data that I want to get is going to be the title so let's get rid of this return here and then let's say cons title and then this is where the structure comes into play you have to know what you're selecting so we're already in the the article with this class of product pod and we want the H3 and in that is a link and that link has a title attribute so why don't we use that and we're just using you know regular JavaScript code here so I'm going to say book Dot and then query selector and I want to select the H3 and then the link inside the H3 and then I want to get the attribute of title okay and then what I'm going to do next is just return title so now let's run web scrape again and see what that gives us and you can see it gives me an array of titles so we we've actually extracted all the titles on the page of course it's only the first page so in addition to the title I want the price so let's get rid of that return let's say cons price and for the price we'll say book do query SEL collector and we have that class I think it's price color so if we look right here we do have a a product price div but in that is two paragraphs and then a paragraph with a class of price color so let's go back into the query selector and let's select the class of price uncore color and then we want the text content within that and then let's just return price and what we should get if we run this next now is an array of prices and there we go so we're able to get the title we're able to get the price let's get the the stock and remember there's a a class of in stock and availability so we can just check for those classes so let's say stock and let's say book. query selector and I want to select the class of in stock and availability okay and then in I just want to return some text that says in stock if it is and out of stock if it's not so what we'll do is just add on a Turner here and say in stock else then we'll say out of stock all right and then let's just do a return of stock and then we should get an array and and you can see that they're all in stock and then the last thing I want to get is the rating and remember the rating actually has a class of um uh let's take a look again here so rating has a star rating class but then it also has whatever it is one through five it'll have that as a class name so what we can do is just simply take the class split it into an array where this is zero and this is one as far as the index and we'll get the the one index so let's go and say const and we'll call this rating set that to book. query selector and we want to select the class of star Dash rating okay they all have that and then get the class name so class name and then I'm going to use split and I want to split by this just a space because there's a space in between both classes and I want the second class which will be the one Index right so if I do a console log of rating and then we run node scrape uh I'm sorry we need to return it not log it uh return rating there we go so now we have just all the the ratings so all we have to do now is is return all of that stuff not just one which is what I've been doing so let's return an object and in that object we'll have the title we'll have the price the stock the rating oh you know what let's get the link as well so up here let's say cons link and set that to book do query selector and we just want to get the so it's the 8 H3 has a link in it so we want the a tag and then I want to get the href attribute so we'll say get attribute and we want the HF okay so that should give us the link so now down here let's just add in link and then we'll console log that let's run node scrape and there we go so we make this a little bigger and you can see now we have uh just a bunch of objects that has all the data for the books so we've successfully scraped that the homepage so now what I want to do is is save that data into a Jason file you could save it into a database whatever you want to do with it so let's import the fs module from FS so this is just the node.js file system module and we're going to go down to the bottom where we have the console log and if you want to still console log it you can I'm not but I am just going to console log and say data saved to and we'll call it books. Json and then right above that we're going to use the fs module and to keep this simple I'm just going to use the synchronous version because we don't really need the asynchronous and this takes in a file name so it's going to be books. Json and then I do want to just run it through the Json stringify method cuz the second argument is what we want to put in that file so let's say json.stringify and we're going to pass in books that's the data then we're going to pass in null for the replacer and two for the the space parameter so if it formats nicely and we'll save that let's come down here and let's run the file so node scrape we see data save to books. Json if we look over here it created a books. Json file and in that is an array with all the data so we could use this for whatever we want if we wanted to you know use it for an an API for our own service our own website we could do that um so yeah I mean that's that's pretty much it that's how you use Puppeteer now at the moment we're only getting the data from this URL which is page one now what if we want more than that what if we want all the data or we want the first five 10 pages whatever so now I'm going to show you how we can handle pagination so first thing let's let's go right under where we create the page variable and we're going to add a couple variables here so the first is going to be called all books which is going to be set to an empty array and at the end of this this will be this will is will be what we actually save to our Json file and then we're going to use let here and we want the current page which is going to start at one and you could start at five or or 10 it doesn't really matter it depends on the data you're looking for but we're going to start at one I'm using let because this will be reassigned it'll be incremented and then let's also add Max Pages which is the number of pages we want to scrape in this case I'm going to say 10 I think there's 50 total but the more pages the more data the longer it takes now the URL we have to change this because again this is only the first 20 books so I'm actually going to change these quotes to back ticks so we can have a dynamic value in here and we want to change it to if we go to the website here yeah so I'm on page two right now so let's just copy that or type it out whatever and I'm going to put that in here except I'm going to change the two to the dynamic value of the current page all right now we need to Loop over the pages and do this stuff down here so let's go right above the URL and I'm going to use a while loop and I'm going to say uh while the current page is less than or equal to the max pages okay so if there's any pages left then we're going to keep on iterating and what we're going to want to put in here is basically everything except for where we save the file so let's grab so basically where this evaluate ends which is right here so I'm going to take that and all this stuff cut it and then put that that into the while loop all right now let's uh let's go still within the while loop so let's see the while loop ends right here I'm going to go still within that and we're going to then take our all books array and we're going to push onto that I want to push the books but keep the ones that are there as well so let's use the spread operator and books okay so every time we iterate through a page we're going to add the books and I I'll even do a console log where we say let's put some back ticks and we'll say books on page and then whatever the current page and then we'll put a colon right here and then just a comma and then books so basically it'll say books on current on books on page two and then it'll show an array of the books on page two you don't have to do this but it's just to to know what's going on and then we'll just take the current page and increment it so we just add one to it and the last thing is down here where we save our data instead of just books we're going to be saving all books and that should do it so I'm going to save and let's go ahead and run node scrape just going to make this a little bigger so we can see what was actually logged all right so you can see it's logging all the objects and we get books this is the last one books on page 10 and then it gives us that array and and let's see we should have right here books on page nine so it's logging it all now let's check and see if it was actually written to the file and if we take a look here we have a lot more than just 20 so yeah so we have 10 pages worth of data and if you wanted to you could you could get all 50 pages but yeah so that's how we can not only scrape data from a page but also use pagination we're using our headless browser to basically you know link click on the link and go to the next page get the data next page Etc and that's it so that's scraping with node and Puppeteer so now we're going to get into using python with beautiful soup so now we're going to scrape this website using Python and beautiful soup so if you watch the node and Puppeteer version then this is very similar it's just using different syntax obviously it's a different language it's a different framework um but the idea is is pretty much the same so let's create uh a new directory to work in I'm going to call this web webscraper Das Python and then I'm going to CD into webscraper Das Python and then from here I'm just going to open up visual studio code and open up my integrated terminal now the very first thing I want to do is create a virtual environment because we're installing packages I don't want to install them globally and I'm assuming you have basic basic knowledge of python if you don't you can check out my python crash course so let's say Python 3-m venv and I'm going to call my environment EnV okay so that will create the environment now we have to activate it so let's say source and then uh EnV SL bin SL activate Okay so that's activated you can see we have EnV in parenthesis here now I'm going to install packages with Pip so the two packages I want to install are going to be requests so I can make HTTP requests in my code and then also beautiful soup uh beautiful soup 4 is the package name I spell that right B yeah okay so now we have those installed so let's now create a file and I'll call this scrape dopy and then I'll open up scrape dopy and the first thing I'm going to do is import what we need so we need requests and then we also want to import from bs4 we want to import beautiful soup and then also um we need the Json package so let's import that all right now I'm going to Define my main function which is going to be my just you know the entry point to our application and then before I do anything there I'm going to run it so we'll say if Dore name Dore equals Dore main then we're going to run main okay and all I'm going to do in main right now is call another function that I'm going to create called Fetch books and we're going to pass in the page number which we'll pass in a one and then right above that let's Define fetch uncore books and that's going to take in uh page number and for now I'm just going to print the page number and make sure oops make sure everything's working so far so then down here let's run Python 3 scrape and we get one good so let's get rid of that print I I want to define a couple variables first is going to be the URL now in the node Puppeteer version I just scraped the first page first and then we added the pagination I'm just going to include the pagination right from the start here which means that for our URL we need to use this which has the slash catalog and then the page number so I'm going to just grab that and then I'm going to use an F string here so F and then some quotes and pass that in and instead of a two that's going to be dynamic and that's going to be our page number okay so that's our URL variable now we need to make the request I'm going to put the response in a variable and we can make the requests with requests.get we're making a get request to the URL okay so that will get the the response and then we want to have a a beautiful soup object let's call this soup set that to beautiful soup and we pass in that response. text and then as a second argument we're going to pass in html. parser and then just to see what that gives us let's go ahead and print soup actually we'll add on to that printify so let's run that so you can see what it gives me is the entire thing all the HTML so now we need to specify the data that we want to extract just like we did in if you watch the the node in putia version so let's get rid of that print and I'm going to create an empty array or an empty list to start with and then ALS o a variable called book elements and from here on that beautiful soup object there's a find all or find underscore all method and that's going to take in basically the wrapper of the content we want which in this case is going to be this article tag with the class of product prod because that wraps around each one of these so let's add in here we want the article tag and then as far as the attributes we want class it's actually going to be class uncore equals and then productor pod now we can use this book elements to basically select what we want from it so I'm just going to close that up and let's do a for Loop here so I'm going to say for book in book elements and then we can select what we want and it's going to be the same stuff that we did with the node version the title the price stock rating and Link so let's do the title first so we can have on that book there's a find method and we can find the H3 and just to kind of look at the structure again so the title is going to be in the article this is right product prod and then the H3 and inside that is an a tag with a title attribute that's where we want to that's how we want to get this title so we'll find the H3 and then from that we can find again and find the a tag and we want the title attribute so this is the syntax we use all right and then let's just do a print of title and we'll run that and you can see now we have all the titles okay next thing I want is the price so let's say price let's do book. findind and it's in a paragraph graph with the class of price color so as a second argument we're going to pass in here class uncore equals and we want price uncore color and we want the text inside so do text which is just like text content in JavaScript and to check that out we can do print price run the file and there we go we have all the prices next is the stock so for the stock I want it to just say in stock if it is or out of stock if it's not there so let's uh let's do this let's say in stock if so in stock if in stock um yeah in stock if in stock in book. find and we want to find the paragraph with the class make sure you put an underscore so class and that has um what is it again so in stock and Avail availability are the classes so in stock and availability right is that formatted right no I have an extra quote here okay so in stock availability and then we want the text within it else then we want to have out of yeah out of stock I think that should do it and let's check that out so I'm going to just print stock and I believe that they're all in stock yep so we just see a bunch of in stocks and then we want the rating so let's say rating and and we want to find so book find and it's in uh paragraph So paragraph and then we want the class class uncore equals in its star- rating now in addition to the star rating class is the actual rating as a class you can see this one has has one so we can basically make this a list and select the second item which would be the one index so let's say star rating and then in Brackets we'll say class and we want the second one which would be the one index so now let's do a print of rating and we should get all the ratings there we go and then the last thing that we want to extract is the link so let's say book. find and that's in the H3 we want to find again again and find the a tag and we want the href attribute so let's print link let's run that and there we go so it doesn't it only has like the the actual route it doesn't have you know it book uh TW scrape or books. what the actual link is if I click on oh here we go yeah so it's slash catalog so it's only giving us this so why don't we prefix it with this all right so let's come back here and let's see we'll do how do I want to do this um you know what we'll leave that as is but then when we append the object will add that beginning part so let's do this let's say books. append and we're going to pass in here an object or a dictionary so title set that to title and then we have what price set that to price and we have stock and rating and we have link so what I'm going to do is set this to an F string and I'm going to paste in what I copied and then we can just say slash and then in curly braces link okay so that will create an array for us and then let's return let's return books all right and we want that return on the same level as the four four which is right there all right cool so well you know what before we return it why don't we just print because I want to make sure that everything is correct before we you know do the pagination and also save it to a file yeah so that looks good and that's the first page so you can see the last book is it's only the Himalayas so if we go to what's this page two let's go to page one and the last book is it's only in the Himalayas and the way we have it right now if we pass in two here we should get we should get the second page so I run that the last book is you you can't bury them all so let's go to the second page and the last book you can't bury them all so we know that this is up to this point is working perfectly so again we do want to um uh we want to return books from here and then we're going to move down to our main function so before we do anything here let's define I'm going to have an array of all books or a list of all books and then our Max Pages again if you watch the node version this is the exact pretty much the exact same thing so now we want to Loop over the pages so let's say four current uncore page in and we're going to use range here you want to rrange one and then maxcore Pages plus one and then within here let's create a variable called books books on page and we're going to set that and this is where we're going to use our fetch books so we can get rid of it from there and let's say fetch books and then pass in whatever that current page is and then we're going to take our all books array or list and let's extend so we're going to call extend and then we're going to add books on page so we're just going through each page and then adding those books to the array and then I'm just going to print just like we did in the node version if you watch that I'm going to say books on page and then the current page and we'll put a a colon and then the books on page so that'll just print it as it as it goes through each page then we want to save the data so let's go on the same level as the four let's say save data to file so we're going to say with and say with open and and with is used to just wrap the execution of a block of code and it just makes sure that resources are are properly managed um you know it guarantees that the file is is properly closed and stuff like that so we're going to say with and then open so open is a built-in python function to open a file and the file that I want to open and it will create if it's not there is books. Json and we want it in right mode so pass in W and I'm going to say as F so basically I'm assigning that to the a variable of f then we're going to use the Json package and a method called dump which does just that dumps the Json into the file and the data we want to dump is all books and um f is the name of the the file okay so basically the books. Json and then I'm going to set the indent to two and that should do it I believe that's yeah I believe that's all we need to do we should do some kind of console log uh after it's done so down here let's just do a print and we'll just say data is is saved to books. Json all right so here we go moment of truth let's go ahead and run it and it's going through and just showing each showing all the books for each page we could make it prettier but it doesn't really matter in the console what matters is a file here and there we go so we got what should be 10 pages of data 10 pages of books all right cool so that's going to do it guys that's how we can scrape a website and handle pagination using both node.js and python so I'll have all the code I'll have both versions in the description if you want that and that's it thanks for watching and I hope you learn something