XPath Crash Course For Python Web Scraping

Key Takeaways

what is going on guys welcome back in today's video we're going to do a crash course on xpath for web scraping in python so let's get right into it [Music] all right so we're going to do an xpath crash course in today's video and we're going to combine this with python web scraping so we're going to learn how to use xpath to do python web scraping using selenium uh so you can watch this video either if you just want to learn xpath in general whatever you want to use it for there are many use cases or if you want to learn python web scraping using xpath because xpath is now the go-to way to extract elements to extract attributes or values from the html code from the html source code using selenium and in general xpath is just a query language you could save for xml for the extensible markup language for xml like files can be also used for html obviously and in this video we're going to first talk about xpath in general i want to give you a crash course on xpath in general and then we're going to apply it to python web scraping so you can use it you can use this video to learn xpath in general or to learn python web scraping with the tool called xpath and for this we're going to start with this website here the books2scrape.com website this is just a web website made for web scraping so you have here a bunch of categories a bunch of books a bunch of buttons you can add something to the basket you can visit the website you can visit the individual product page you can get some information here and we're going to also use the tool called xpather.com now there are a lot of similar tools you don't have to use this one essentially what we have here is we enter some html code down below here and then we can use a query and xpath query to find certain elements so what we're going to do now is we're going to go to books2scrape.com i'm going to right click i'm going to go to fuse so view source code i'm going to copy all the source code and i'm going to paste it into the xpather down here and then i can specify a simple query and we're going to start with a very simple query you can see here we have a bunch of different elements we have uh the head tag and then we have the body and inside of the body we have a header we have a bunch of diff boxes with different classes and we have a lot of uh where are they articles right so we have when we scroll down a lot of articles um and inside of the those articles we have diffs again and we have links so anchors with ahref uh we have paragraphs we have also somewhere lists and list items and so on and we can just select certain elements so let's start with the most basic query here we can just do something like give me all the articles that contain an anchor that contain a link and from that link give me uh for example what do we have i think here we're not at the articles uh the link usually has let's scroll down to an article there you go article link um some of the links have a title there you go here you have an h3 which is a heading three you have an a tag in here and you have um let me just zoom in a little bit so that you can see what we have here uh and you have also the title attribute so what can we do here in xpath is we can specify slash or slash slash now slash basically means it has to be a direct child of whatever you're selecting right now and slash slash means it has to be somewhere below it so if i say article i will get all the articles selected so here you can see to the right i have all the selections all the things that match the xpath query in this case slash slash article and if i click on one of them you can see okay i have this article here that is matching the query and now i can say either slash or slash slash so if i say just slash it's going to look exactly below the article so you can see here the article has a div box direct below directly below it uh or a paragraph directly inside of it but it also has an anchor tag that is not directly inside of it it's inside of the div box that is inside of the article so if i say slash a it's not going to find anything if i say slash diff it's going to find a div because it's directly below the article or directly inside of the article but if i say slash slash a it's going to find the a because it's somewhere inside of the article but it's not directly inside of it so slash again i repeat this a couple of times slash is when it's directly the next thing you enter the article and you can find it directly at this level inside of the article then you use one slash and if you say slash slash you get anything that's somewhere in there so if i say slash img i'm also going to find this image here even though it's inside of an a tag that is inside of a div box that is inside of the article right because it's somewhere in there if i just say slash i'm not gonna i'm not gonna find it so we're gonna say okay give me the article give me all the links that are inside of the article and then we can say give me the attribute called title and of course this will not apply to all the anchor tags because here for example we don't have a title but here we do have a title and in order to specify attributes what we do in xpath is we use the at symbol so the mail symbol so to say which is the this decimal here and if i want to access the attribute off a certain item off a certain element i just say slash and then add at title in this case i get now all the titles from all the anchor tags that are inside of articles that have a title right so here i don't have a title which is why i don't get a title obviously but you can see i have a bunch of titles and i get them here as a result set so you can see that xpath is a very nice language that can express we can express certain things without having to go through loops i don't have to say okay get all the elements by tag article now from that get all the childs get all the elements a get all the attributes uh and see if there is an attribute called title get the value and so on we can just say slash article slash slash a slash at title and we get all the titles of all the links that are inside somewhere of an article that is quite a simple query now one thing that i want to show you here it doesn't change anything about the query uh is we can also say dot slash title uh because dot is essentially referring to itself now in this case it's useless in some cases we will see it can be quite useful but essentially what this says is give me all the articles give me all the uh the anchor tags that are somewhere inside of those articles and then give me those anchor tags again so the point is referring the dot is referring to the a itself so in this case it's just redundant it doesn't do anything but sometimes if you want to refer to something itself you use the dot so that is quite simple let's um move on a little bit what we can also do is we can move up so we can say for example okay give me all the articles give me all the a's and then go up one element so give me all the elements this is now how do you phrase this you get all the articles you get all the anchor tags inside of those articles and then you don't get those anchor tags um i think yeah you don't get those anchor tags but you get the item that is above them so in this case i select all the articles i get the anchor tag but i don't get the anchor tag itself i go one level up in this case this is a div box here and i get this particular thing so i'm selecting this div box here or in the case of this link i'm selecting this heading here because that is the parent element and i can then say okay give me the add class of that and you can see in this case the h3 never has a class but the div box always has the class image container and you can see i get a result set here with the class names image container every time because all the div boxes have the same class so with dot dot or with point point or period period uh you go up to the parent element um now we can also do something else we can say okay give me all the classes in general so i can say okay give me from all the elements that have an attribute class give me the class name and then you can see here i have a bunch of different classes a lot of them are repeating so what i could do is i could take those in python and make a set of them to get all the classes that exist this is one possibility what we can do with that now what we can also do is if we're not interested necessarily in um in the class itself but we want to know that this div box does have a class uh or that this anchor tag for example does have a title we can do the following thing so i can say okay give me all the links somewhere so i get all the anchor tags but now from all those here exclude those that do not have a title or in other words get only those who do have a title now of course i can just say add title this works but maybe i don't want to have the title maybe i just want to select those links that do have a title but i don't want to have the title as a return value in this case i can use square brackets here and say add title inside of them so the result is the full anchor tag but only for those anchor tags that do have a title so you can see i'm selecting the whole anchor tag not just a title but those anchor tags here are not selected because they do not have a title so i can go here and say href at ahref to get the links of all the anchor tags that do have a title i don't get the title i get the href but they have to have a title because if i remove that i'm going to get way more hrefs because i'm going to get those here as well so that is one thing we can do it's called a predicate if you put something in square in square brackets and it can be used for conditions so what we can also do is for example we can get all the anchor tags that do have some note inside of them so i can say okay give me all the anchor tags would be this one and give me all the anchor tags that do have something inside of them would be this one here so it would be all the anchor tags that do have something else in this case an image tag inside of them those would not uh those would not fulfill this condition because they only have text so in this case we say give me all the anchors that do have something in them that is not just text or we could do to get these things in general so in this case we don't just say um give me all the anchor tags that do have something inside of them but we say give me that thing inside of them uh give me all the things that are inside of anchor tags directly or we could also say slash slash but in this case it doesn't make a difference because we don't have uh multiple levels uh what else can we do we can say if we want to keep it more general we can say okay give me all the things in general give me all the notes that contain any attributes so this would be like that we would say slash star for everything that has some attribute this means any attribute at any attribute um and in this case we would get everything that has any attribute is there anything that is not selected i mean we'll we won't see it here because the whole i think the whole uh where is it the whole body has attributes so everything's selected but um yeah you can see also the head is probably not selected and the title is not selected but they are selected because of the html right so if i remove this i think it should not be selected there you go so now it doesn't have any attributes and because of that it's not selected so that is one thing that you can do um now we can also go with partial classes so what what do we do for example um if we want to say that um that we want to filter for specific attribute value so we want to say for example give me all the diff boxes not just they they don't just have to have a class we want to also have a specific value so i can say okay give me all the div boxes that have a class like this so this is the condition but also i want this class to have a specific value namely row for example then i can do it like this so in this case i select all the class all the divs that have a class equal to row i don't select all the divs that have a class equal to page inner for example so this works and um sometimes you will notice that you have a problem you might want to have um i think we should have somewhere here let me just find them um do we have coal sm8 should be the thing that i'm looking for there you go so here for example you have um the diff box that has the following class call sm so co column small eight and h1 as a class but those are two different classes so in css those would be two different classes that we need to select and maybe i don't want to look for the exact class but i want to find all the cosm8 but this would not work like this because the problem is that when i'm using the equal sign here i'm asking for the exact uh for the exact classes so for everything if i want to say okay it has to contain this class but it can also have something else but i'm only focusing on this here so if it only contains that it would be selected but it contains if it contains that and something else it will also be selected uh i have to use a different function i have to use the contains function here so i would say okay all the diffs work contains and then add class and co sm h is true in this case this would select this one and it would also select this one that does not contain an h1 right because it contains this string inside of the class attribute so this is how you look for partial classes um also you can look for text so for the content of the thing so for example if i want to select this one here i would do give me all the a's where the text is something so i can say um [Music] what would we do we would say text is equal to i mean i don't think that i'm going to match this exactly so let's do something else let's say we want to see any element that contains a certain text so i can say okay give me any element that contains the text books right so how do we do that we don't want to look for the exact text we want to find a contained text how do we do that we say okay this is a little bit more complicated now and now you might understand why we need the dot operator because what we're doing now is we're saying okay give me all the elements get the text of this element and then apply the contains function onto it so we use another square bracket operation another predicate here let's close them up and we say okay we want to select the text and for this text the following has to be true it has to be true that it contains uh and what contains it the text itself so period contains the text book so yeah and this is how we find these elements we find any element that contains the text book so again we select all the elements and we only want to select those elements where the text contains book or we can also say books or we can say novel and then we're going to get something else we're going to get the element down here and if i change this now to a we're still going to get the same thing um so this is how you do that you say okay the a the anchor tags but we have a text in there and for this text the following has to be true that the text itself this is why we use the dot operator contains novel or novels for that matter right so this works um what else can we do we can say we can do also some fancy stuff like counting elements we can say okay give me all the elements um where there are items inside of that element that are that contain exactly 20 list items for example so we can say give me all the elements and then we can say count and then we say dot so from the current element all the lis that we find this number has to be equal to 20. and we close this as a condition here so in this case we find first of all let me show you that we can we can find in general uh something that contains list items so all the elements that contain list items would be this one here a lot of them but we can say okay count the list items that occur after the current point this number has to be equal to one two three we can set whatever number we want uh want i think for two we had some results uh maybe for do we find another number that produces results no but for 20 we will find some um and here we get the result you we have one element the ordered list with a class row where we have exactly 20 list items and we can say i don't want to have the element itself i want to have those 20 list items there you go and this is how you get those so this is a little bit more fancy then another thing we can do is we can filter for ancestors siblings and other relationships now for the sake of completion i recommend you go to some documentation or some tutorial where all the different um axes are listed i think they're called axes but i'm going to show you two of them i'm going to show you the following sibling and i'm going to show you the ancestor because what you can also do is not just ask for children or for parents you can also say something like um i wanna have a diff box that comes after another diff box and how do you do that you say okay i wanna have all the diff boxes that come after so you select all the diff boxes and you then select all these siblings of those diff boxes so you say following dash sibling and then colon colon diff so in this case we get this one here because it comes directly after or not necessarily directly because it comes in general after this div box but you can see that this div box here is not selected because there is no div box above it in the same thing so yes it's a child of a div box and it also is inside of another diff box but in its scope inside of this div box here we don't have another div box that comes before this div box here and i think we should also have something like preceding sibling which is the exact opposite you so you want to have a diff box that comes before another diff box um so that is the opposite but you can do it like that and you can also say i want to have the one diff box the first div box that comes after a diff box so in this case i think we have this one here um or let's do it manually let's say i have another div box here diff diff so this div box is also selected now because it comes after this div box and this one is also selected because it also comes after this div box um this diff box here but also because it comes after this one um i think if i say one uh no this is actually a bad example let me do it in a different way how can we do it [Music] let's go with diff box where the class is equal to row now actually to where the id is equal to promotions left promotions left there you go um and now let's remove the one so this this illustrates the example okay oh now i change this come on i changed this now okay let's write it again diff where the id attribute is equal to promotions left and here we want to find the diff box uh the following sibling that is the div box okay so this should illustrate the example quite well i have this div box here and this has the id promotions left then all the siblings all the following siblings that are diff boxes are all these diff boxes that come afterwards however if i now say in square brackets one i'm only gonna get the next one so i have this id promotions left div box here and i have two siblings two following siblings but i'm only interested in the first one so this one is no longer selected i think this should also work with the second one third one and so on we don't have a third one in this case uh but you can see how you can also influence that and then uh you can do the same thing uh with an ancestor so we can say for example give me all the diff boxes that have uh an ancestor and sister article and in this case we get all the diff boxes here that are inside of an article right so this is also something you can do and last but not least let's do a little bit more complicated stuff for the last query before we go into the python code we're going to do the following query it's going to be a query that finds all the titles and prices of books that have a one star rating so in this case let's let's write the query and then i'm going to explain it i think that's um that's more reasonable than to explain it while writing it i'm going to say give me all the articles articles where there is a p text or paragraph um and i have prepared the statement this is why i know um what we need to look for but essentially we have the star rating one class inside of the p tag this is what we're going to look after so or look for we're going to say class equals star dash rating and then one give me all the articles um that have such a p tag inside from those articles give me the h3 tag so the heading and from that heading give me the a tag and from that a tag give me the title attribute so those are the six books that have a one star rating on this website and now i'm going to combine this with a so-called pipe operator with this one here and i'm going to say okay give me now the article p or actually this is going to be the same thing up until the point of this here so we're going to copy this so also star rating one but this time we don't want to have the heading we want to have uh another paragraph inside which will have the class being equal to price underscore color and from that we want to get the text all right so what we're doing here is um we're getting all the articles that have a p tag inside of them that has the class star rating one this is those are the uh those are the articles that have us a rating of one from those articles that we select we now get the h3 tag which is the heading inside of that heading we have a link and this link has a title which is the book title in the same article that also has this p with star reading one we also have another paragraph that has the class price color that gives us the price so as a result of that query and this pipe symbol here combines the two results so it connects them we have now the title and the price as a result as you can see so those are the books that have a one star rating and this is also the price of the book and we can change this i think to two and two two here and now we have all the two star ratings and all the prices of the books that have a two star rating so now let's go ahead into python and apply all this to web scraping using selenium and for that obviously we're going to have to install selenium by saying pip install selenium and i think we also need to install pip install web driver dash manager in order to be able to use a chrome driver then we're going to save from selenium dot webdriver import chrome i think recently i had a tutorial on uh selenium website automation where i also used xpath so you might want to check that out for more details on selenium in general this is going to be more focused on xpath but it's still going to be quite simple for beginners uh from selenium.webdriver.chrome.service we're going to import the service we're going to also say from selenium.webdriver.chrome.com we're going to import options and then from webdriver manager.chrome we're going to import the chrome driver manager so all of this is going to be just for yeah for creating the chrome application the chrome web driver here we're going to say now the url that we're going to target is going to be the same one https colon slash books.2scrape.com and um then we're going to say now then we're going to say that we want to define new options we're going to say chrome underscore options are going to be equal to options and i'm going to just add the option that we don't close the browser uh when it's done so we want to keep it open up just so we can analyze a little bit what's happening and in order to do that we say chrome options add experimental option and we want to add the detach option and we want to set it to true then we say driver equals chrome we're creating a new chrome driver with a service being equal to and now we create a new service instance the service instance service instance is going to be based on a chrome driver manager um and we're going to call the install function on it and then we're going to say options equals chrome options so this is the whole setup now all we need to do is we need to say driver dot get url and then we're also going to say driver.maximize window and when i run this now it should install the chrome driver and once it's installed it should open up the browser and you can see now chrome is running you can see chrome is being controlled by automated test software and it just navigated to the page and now we can use xpath to do whatever we want so we can take the last query and um instruct selenium to do it how do we do that we say results equals and then we say driver.find element and there are multiple ways to find elements but the go-to way nowadays is to do it with xpath so we provide xpath here as a string to know that we're applying x xpath and then we can state the query that we had so let's define the x path on this underscore query and this is going to be the query that we used um that we used last so it's going to be the find the article that has a paragraph with a class being equal to star dash rating one and from that get me the h3 from that get me the a tag now one thing that's important here is we're calling the find elements function so instead of saying now slash at title to get the title we're just gonna go for the a tag for the anchor tag and then in python we're going to extract the attribute from that element because this function can only handle elements this is why we don't say slash at title we just get the a tag and then we're going to get the title from the a tag and i'm going to combine the still with slash slash article where we have a p tag with a class being equal to star dash rating one um and then we're going to say again p that has a class being equal to price underscore color and that is essentially it again we're not going to go for the text we're not going to go for slash text we're going to extract this with python and here we're just going to say now that we want to send the xpath query and then we're going to handle the results by saying for result in results if the result dot tag name is equal to a so if we get an anchor tag we're going to print the result dot get attribute title so we want to get the attribute title otherwise uh in this case it's going to be uh what was it it's going to be p tag so you can also say elif result.tech name equals p in this case an else is enough we're going to print the results.getattribute and in order to get the text we now say oh why is this a string in order to get the text we now say inner html like this this is going to get us the text so i can run this now it's going to automatically i'm not doing anything it's going to open up the website and now if i go into pycharm you will see that we have the title and the price of the individual books so that's it for today's video i hope you enjoyed it and hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free other than that thank you much for watching see you next video and bye [Music] you