DEV Community

Discussion on: If you would need to scrape many different websites nowdays, which tool/language combo would you pick?

talha131 profile image
Talha Mansoor

I do not need to interact as a human, but just collect news articles from different websites, at scale.

If it is scale you are looking for then best option would be with Scrapy Cloud. You can also run multiple Scrapy spiders in a process.