Python Tutorial: Crash Course in XPath

Key Takeaways

at this point we've run through the basics of HTML with some wordy ways to describe how to navigate to particular elements however if we want to describe where these elements are within our programs programs made to navigate and scrape HTML then we need to build up a standard program friendly language or syntax to do so you have noticed that all your exercises at this point have been multiple-choice and this is because we didn't have the necessary understanding of how to turn our wordy navigation of HTML into a variable for the computer to ingest that changes now this lesson will give a crash course in some basics of what's called XPath notation one of two common choices for this purpose and in the next chapters we will go deeper into both syntaxes with many more examples jumping right in a simple XPath string we could write in Python is given here one nice property of XPath notation is that you might already have some familiarity with similar syntaxes because it uses a single forward slash in an analogous way as you do when you are navigating directories or typing a URL into your browser the single forward slash moves us forward one generation in fact if we think of the tag names as the directory names then these simple X paths will look very much like navigating between directories what might seem unfamiliar are the brackets these brackets are used to help specify which element or elements we want to direct to for example there could be several div elements which are children of the body element that is several div siblings so we can use the brackets to narrow in on the div element we want to illustrate the sample XPath string we wrote in the last slide here we have highlighted the div element which would be selected within a tree representation of some HTML notice that the number two in the brackets of our XPath expression refers to the second div element of the three div elements ordered from top to bottom as usual paying attention to the fact the first child of the body is a span element so is not counted when looking at the div elements another important feature of XPath notation is the double forward slash using the double forward slash tells us to look forward all future generations instead of one generation like the single forward slash so for example we could navigate to all table elements within an HTML document by simply typing double forward slash table or we could want to restrict to a specific div element say the one we learned how to navigate to in the last couple slides and navigate to all table elements which are descendants of that div element and that's it for now we've only just scratched the surface of XPath notation but we've gone deep enough that you can begin to write some code and get your feet wet navigating HTML computationally let me emphasize here that XPath is general meaning that it is not Python specific so if you decide to start scraping the web in our say most libraries there will also be able to read and interpret your