Ethics of Data Harvesting: Configuring robots.txt and User-Agent to Bypass the Ban-Hammer

Web scraping is often characterized as a cat-and-mouse game, a technical arms race between those who...