Ever had to scrape a website that blocks requests after a few tries? I built a simple Python script that rotates user agents and proxies to avoid getting banned. It uses a list of fake user agents and proxies from a free API. Here's the core loop:
python
import requests
from itertools import cycle
user_agents = ['Mozilla/5.0 (Windows NT 10.0; Win64; x64)...', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...']
proxies = ['http://proxy1:8080', 'http://proxy2:8080']
ua_cycle = cycle(user_agents)
proxy_cycle = cycle(proxies)
for url in urls:
headers = {'User-Agent': next(ua_cycle)}
proxy = {'http': next(proxy_cycle), 'https': next(proxy_cycle)}
response = requests.get(url, headers=headers, proxies=proxy)
# process response
This works for small projects, but for heavy scraping I've been using a dedicated proxy service. What's your go-to method for avoiding IP bans?
Top comments (1)
Congrats on the quick traction! It's always inspiring to see a real-world itch-scratcher turn into revenue. The LinkedIn comment strategy is a great reminder that distribution doesn't need a big budget, just genuine engagement. What's been the most surprising feedback from your first 80 users?