DEV Community

loading...

Discussion on: Building a Polite Web Crawler

Collapse
turnerj profile image
James Turner Author

Yep, you can supply any user agent in the crawl settings (see example). Providing a user agent like that, while will work perfectly fine, circumvents sites giving direction into what content is accessible or not via the "robots.txt" file.