Building a Resilient Web Scraper with Retries and Proxy Rotation

#python #webscraping #proxy #webdev

Most scrapers work in the demo and die in production. The difference is almost never the parsing logic, it is everything around it: the request that times out, the one that comes back as a captcha, the address that gets blocked at hour two of a six hour run. A resilient scraper treats failure as normal and keeps going. This post is a practical structure for that, built around two ideas that carry most of the weight, retries and proxy rotation.

Failure is the default, not the exception

At any real scale, a percentage of your requests will fail for reasons that have nothing to do with your code. Networks drop packets. Sites rate limit. A proxy address gets flagged mid run. If your scraper treats the first error as fatal, it stops on problems that would have solved themselves on a second attempt from a different address. The goal is a loop that expects failure, retries intelligently, and only gives up after it has genuinely tried.

Retry with backoff and a fresh address

The core pattern is a retry wrapper that catches failures, waits a little longer each time, and crucially retries through a rotating endpoint so each attempt leaves from a new address.

import time, random, requests

PROXY = "http://user:pass@proxy.host:port"
proxies = {"http": PROXY, "https": PROXY}

def get(url, retries=4):
    for attempt in range(retries):
        try:
            r = requests.get(url, proxies=proxies, timeout=20)
            if r.status_code == 200:
                return r
            if r.status_code in (403, 429):
                raise requests.HTTPError(r.status_code)
        except requests.RequestException:
            pass
        time.sleep((2 ** attempt) + random.random())
    return None

Two details matter. The backoff grows so you are not hammering a struggling server, and the jitter keeps many workers from retrying in lockstep. Because the proxy rotates per request, each retry lands on a different exit, so a 429 caused by one address does not repeat on the next attempt.

Separate transient failures from real ones

Not every error deserves a retry. A 404 is a real answer, retrying it just wastes time and addresses. A 429, a timeout, or a connection reset is transient and worth another attempt. Encode that distinction so your scraper spends its retries where they help.

Retry: timeouts, connection errors, 429, 502, 503, and challenge pages.
Do not retry: 404 and 400, which are genuine responses about the URL itself.
Stop the whole job: repeated 401 or auth errors, which mean your credentials are wrong, not your luck.

Rotation is what makes retries work

Retries without rotation just replay the same failing request from the same flagged address, so you burn attempts and get nowhere. Rotation is the other half: each retry, and ideally each request, leaves from a fresh address, so the per address rate counter never climbs and a single flagged exit never stalls the run. This is where the address pool behind your endpoint decides whether the whole structure holds.

For that you want private addresses that are yours alone, not a shared pool someone else already burned. WinGate provides private IPv4 proxies with SOCKS5 and built in rotation from a worldmix pool, so every retry can land on a clean exit without you managing addresses by hand. The rotating pool cycles automatically, traffic is unlimited so long runs do not meter you, and it handles up to 5000 threads for wide concurrency. It speaks HTTP, HTTPS, and SOCKS5, so it drops into the retry wrapper above by swapping the endpoint.

An honest note: retries and rotation make a scraper survive normal failure, they do not make it welcome. You still owe the target sensible pacing, and no proxy exempts you from a site's terms. There is a free 2 hour test, so point this structure at your own targets and watch how the success rate holds across a long run before you commit.