Shell Tutorial: Advanced downloading using Wget

Want to learn more? Take the full course at https://learn.datacamp.com/courses/data-processing-in-shell at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- So far, we've learned how to install and do basic file downloads using either curl or Wget. In this lesson, we will focus on getting the most out of Wget by going over more advanced techniques for data downloading. A common way for data people to handle multiple file downloads is by storing the file locations in a file and pass that meta file to the downloading program like Wget. In this case, all the URLs for the files we want to download are stored in the file URL underscore list dot txt. Let's use the cat command to print and preview the URLs really quickly. After confirming that the URLs are indeed stored in this file, we can now pass this file to Wget. Note that we need to preface this with the dash-lowercase-i option flag, so Wget knows that we are reading URLs from a local or external file. The command reads: wget dash lowercase-i url underscore list dot txt Finally, it's worth noting not to insert any option flags in between the dash-i and the URL file. If other option flags are needed, put it before dash-i. Sometimes, it's useful to make sure Wget does not consume your entire bandwidth with the file download. You can set an upper download bandwidth limit using the dash-dash-limit-rate option. Set the limit rate equal to a whole number, which will automatically convert to bytes per second. For example wget dash dash limit rate equal to 200k will make sure your download rate will not exceed 200 kilobytes per second as you download the files saved in the URL list. For downloading smaller files, enforcing a download bandwidth won't work as well. To avoid overtaxing the file hosting server, it is more useful to enforce a mandatory wait time between file downloads using dash-dash-wait. The default time interval is s