Free Proxy Rotation for Web Scraping: When You Need It and How to Set It Up

#security #webdev #javascript #tutorial

Most scraping doesn't need proxies. But when it does, here's how to set up rotation without paying for proxy services.

When You DON'T Need Proxies

Using public APIs (Reddit JSON, GitHub API, etc.)
Scraping < 100 pages from one site
Using proper delays (3-5 seconds between requests)
Site doesn't block by IP

When You DO Need Proxies

Scraping 1,000+ pages from one site
Site uses IP-based rate limiting
Need geographic diversity (prices vary by country)
Site blocks datacenter IPs

Free Proxy Options

1. Apify Proxy (Free Tier)

Apify includes proxy with their free plan:

const { Actor } = require('apify');

Actor.main(async () => {
  const proxyConfiguration = await Actor.createProxyConfiguration();
  const proxyUrl = await proxyConfiguration.newUrl();

  const res = await fetch(targetUrl, {
    agent: new HttpsProxyAgent(proxyUrl)
  });
});

2. Tor Network (Free, Slow)

# Install Tor
brew install tor
tor &

# Use in Node.js via SOCKS proxy
# Tor runs on socks5://127.0.0.1:9050

3. Rotate User-Agents Instead

Often, rotating user-agents is enough without proxies:

const agents = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64)',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)',
  'Mozilla/5.0 (X11; Linux x86_64)'
];

const randomAgent = () => agents[Math.floor(Math.random() * agents.length)];