This post was originally published on this site
We help our customers power their job boards by scraping jobs from around the internet. But we don’t scrape job boards without permission. One question we hear a lot when we’re explaining our offerings boils down to this: Why not?
In this blog post, we’ll answer that question – and hopefully give you a new appreciation for the ethics of web scraping and what it means to be a good citizen of the internet.
Background: What Is Web Scraping?
If you’re new to the world of web scraping, here’s your primer: web scraping is the practice of pulling data from websites via code. If you use the internet, you almost certainly benefit from web scraping on a daily basis because search engines function by scraping other websites.
Google, maybe the most famous web scraper, uses robots called spiders to scrape websites and populate them as links on results pages. When you type “best job board software” into Google, the links you see are there because Google scraped websites looking for signals that those websites had information about the “best job board software”
Obviously, this is an incredibly valuable example of web scraping: without services like Google, it would be much harder
