We're a place where coders share, stay up-to-date and grow their careers.
I do not need to interact as a human, but just collect news articles from different websites, at scale.
If it is scale you are looking for then best option would be scrapy.org/ with Scrapy Cloud. You can also run multiple Scrapy spiders in a process.
If it is scale you are looking for then best option would be scrapy.org/ with Scrapy Cloud. You can also run multiple Scrapy spiders in a process.