Shell Tutorial: Downloading data using curl

Want to learn more? Take the full course at https://learn.datacamp.com/courses/data-processing-in-shell at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Welcome to Intermediate Shell! My name is Susan Sun, and I do data work. I'm looking forward to learning with you in this course. In data, many of us bypass the command line in favor of GUI interfaces like Anaconda and RStudio because that is what we are familiar with. However, taking the time to learn data science on the command line is a great long term investment that will, ultimately, make us better and more productive data people. In this course, we take a practical approach and learn command line tools useful for everyday data processing and analyses. First, let's learn how to download data files using curl. curl, short for Client for URLs, is a Unix command line tool for transferring data to and from a server. It is often used to download data from HTTP sites and FTP servers. To check if curl has been properly installed, type the following in the command line: man curl If curl has not been installed, you will see: curl command not found To install curl, follow this link. If curl is installed, your console will look like this: Keep pressing Enter to scroll through the curl manual. To exit and return to your console, press q. The basic syntax for curl has the following structure: curl, option flags, URL The URL is required for the command to run successfully. curl supports a large number of protocol calls. For a full list, use curl dash-dash-help. Let's download a single file stored at this hypothetical URL using curl. To save the file with its original name datafilename-dot-txt, use the option flag dash-uppercase-O. This reads: curl dash uppercase-O followed by the file URL location To save the file under a different name, replace dash uppercase O with dash lowercase o and the new filename.