DEV Community

Discussion on: DOs and DON'Ts of Web Scraping

Collapse
 
anderrv profile image
Ander Rodriguez

Hi, thanks!!

It's an overkill if your are sure that they won't block you. But depending on the sites, you cannot be 100% sure.

By user_agents you mean the library. As far as I know, it will detect capabilities based on an User Agent string. For our case, we would need a generator.