DEV Community

Build a simple python web crawler

pranay749254 on August 20, 2017

What is a Web Crawler? Web crawler is an internet bot that is used for web indexing in World Wide Web.All types of search engines use we...
Collapse
 
adamant11 profile image
Adamant11

In addition to scraping, you do need a proxy in most of the cases. Smartproxy seems to have the best cost to quality ratio at the moment. Are you covering your proxies?

Collapse
 
glutonion profile image
Comment marked as low quality/non-constructive by the community. View Code of Conduct
GlutoNion

Any of the residential services does work alright for scraping. Have tried Smartproxy and Luminati - both are quality. However, Smartproxy is a lot cheaper, Luminati has a higher IP pool.

Collapse
 
lukaszkuczynski profile image
lukaszkuczynski

Great, scraping is so great with python. Have you ever been wondering about using something like scrapy from here