Python Tutorial : HTTP requests to import files from the web

Key Takeaways

Congrats on importing your first web data in order to import files from the web we use the URL retrieve function from URL Lib requests let's now unpack this a bit and in the process understand a few things about how the internet works URL stands for Uniform or universal resource locator and all they really are are references to web resources the vast majority of urls are web addresses but they can also refer to a few other things such as file transfer protocols ftp and database access will currently focus on those urls that our web addresses or the locations of websites such a URL consists of two parts a protocol identifier HTTP or HTTPS and a resource name such as data count comm the combination of protocol identifier and resource name uniquely specifies the web address to explain URLs i've introduced yet another acronym HTTP which itself stands for hypertext Transfer Protocol Wikipedia provides a great description of a TTP the hypertext Transfer Protocol is an application protocol for distributed collaborative hypermedia information systems HTTP is the foundation of data communication for the worldwide web note that HTTP is a more secure form of HTTP each time you go to our web site you are actually sending an HTTP request to a server this request is known as a get request by far the most common type of HTTP request we are actually performing a get request when using the function URL retrieve the ingenuity of URL retrieved also lies in the fact that it not only makes a get request but also saves the relevant data locally in the following you'll learn how to make more get requests to store web data in your environment in particular you'll figure out how to get the HTML data from a web page HTML stands for hypertext markup language and is the standard markup language for the web to extract the HTML from the Wikipedia homepage for example you import the necessary functions specify the URL package the get request using the function request send the request and catch the response using the function URL open this returns an HTTP response object which has an Associated read method you then apply this read method to the response which returns the HTML as a string which you store in the variable HTML finally you remember to be polite and close the response now we're going to do the same however here we'll use the requests package which provides a wonderful API for making requests according to the requests package website requests allows you to send organic grass-fed HTTP requests without the need for manual labor and the following organizations claim to use requests internally Her Majesty's Government Amazon Google Twilio NPR Obama for America Twitter Sony and federal us institutions that prefer to be unnamed moreover requests is one of the most downloaded Python packages of all time pulling in over 7 million downloads every month all the cool kids are doing it let's now see requests at work here you import the package requests specify the URL package the request send the request and catch the response with a single function requests get and finally you apply the text method to the response which returns the HTML as a string ok that's enough out of me for the time being let's get you hacking away at pulling down some HTML from the web using get requests get coding