Python Tutorial : Importing flat files from the web

you're now able to import data in Python from all sorts of file types flat files such as txt s and CS fees other file types such as pickled files Excel spreadsheets and MATLAB files you've also gained valuable experience in querying relational databases to import data from them using sequel you've really come a long way congratulations however all of these skills involve importing data from files that you have locally much of the time as a data scientist these skills won't be quite enough because you won't always have the data that you need you'll need to import it from the World Wide Web say for example you want to import the wine quality data set from the machine learning repository hosted by the University of California Irvine how do you get this file from the web now you could use your favorite web browser to navigate to the relevant URL point and click on the appropriate hyperlinks to download the file but this poses a couple of serious problems firstly it isn't written in code and so poses reproducibility issues if another data scientist wanted to reproduce your workflow she would necessarily have to do so outside Python secondly it is not scalable if you wanted to download 100 or 1000 such files it would take 100 or 1000 times as long respectively whereas if you wrote it in code your workflow could scale as reproducibility and scalability are situated at the very heart of data science you're going to learn in this chapter how to use Python code to import and locally save datasets from the World Wide Web you'll also learn how to load such data sets into pandas dataframes directly from the web whether they be flat files or otherwise then you'll place these skills in the wider context of making HTTP requests in particular you'll make HTTP GET requests which in plain english means getting data from the web you'll use these new request skills to learn the basics of scraping HTML from the internet and you'll use the wonderful Python package beautifulsoup to parse the HTML and turn it into data now there are a number of great packages to help us import web data herein you'll become familiar with the URL Lib and request packages will first check out URL Lib this module provides a high level interface for fetching data across the world wide web in particular the URL open function is similar to the built-in function open but accepts universe saw Resource locators URLs instead of filenames let's now dive directly into importing data from the web with an example importing the wine quality data set for white wine don't get jealous in the first interactive exercise it will be your job to import the red wine data set all we have done here is imported a function called URL retrieve from the request sub package of the URL Lib package we assign the relevant URL as a string to the variable URL we then use the URL retrieve function to write the contents of the URL to a file wine quality white dot CSV that's it now it's your turn to do the same but for red wine in the following interactive exercises you'll also figure out how to use pandas to load the contents of web files directly into pandas dataframes without first having to save them locally happy hacking