DEV Community

Discussion on: WebScraping [Part-1]

 
mefaba profile image
AW A RE

Any public data is legally scrapable by law. And it should remain legal.

Thread Thread
 
cubiclesocial profile image
cubiclesocial

Terms of Service are unsigned contracts unless you sign the contract by doing something like create an account and agree to the ToS. Then contract law may apply and "no scraping" clauses in such contracts might be legally binding. I'm not a lawyer but the law is a lot more complex than you think and each region of jurisprudence is different in how it applies its own laws. Your blanket assertion that scraping anything published on a website is legal is false. If someone has to login to obtain content (i.e. agree to a ToS) or they knowingly obtain content that is known to be sourced via illegal means, then civil or even criminal actions can be taken against that person.

Legal issues aside, web server operators can also block those who make excessive requests to their servers. IMDB has official data dumps of their database. It's not perfect since some information is missing but it is a good enough starting point for most purposes. Since IMDB makes data dumps available for direct download and is more efficient than scraping, IMDB has every right to block anyone scraping their main website.