Are you getting IP blocked repeatedly when web scraping at scale?
Is this happening even after passing it through a couple of free proxy servers?
It might be because the target website’s algorithm might be picking up on who you are by the User-Agent-String signature that your curl request or any other library you might be using is sending. Even the absence of it is telling.
Top comments (0)