Scrapy Middlewares for Proxy Rotation and Retry

Scrapy is built for crawling at scale, and that is exactly why a naive setup gets banned so fast. Out of the box, every request leaves from one address, and a Scrapy spider can fire hundreds a minute. The target sees the burst and blocks you. The clean fix lives in the middleware layer: a proxy middleware to route requests through a rotating pool, and a retry middleware to recover the ones that still fail. Here is how to wire both.

Where proxies belong in Scrapy

Scrapy processes every request through a chain of downloader middlewares. That is the right place to attach a proxy, because it applies to every request without touching your spider code. The simplest version sets proxy on each request meta.

class ProxyMiddleware:
    def __init__(self, proxy):
        self.proxy = proxy

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("PROXY_URL"))

    def process_request(self, request, spider):
        request.meta["proxy"] = self.proxy

With a rotating endpoint you point PROXY_URL at one address and the pool changes the exit per request, so you get rotation without maintaining a list yourself.

Enable it in settings

Middlewares only run when registered. Add yours to the downloader middleware dict and set the endpoint.

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}
PROXY_URL = "http://user:[e-mail]:8080"
RETRY_TIMES = 4
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]
CONCURRENT_REQUESTS = 32
DOWNLOAD_DELAY = 0.5

The number after each middleware is its order in the chain. The proxy runs before the retry, so a retried request also goes out through the rotating endpoint and lands on a fresh exit.

Retries and rotation together

Scrapy's built-in RetryMiddleware already handles the retry logic. The important part is that it cooperates with rotation: when a request fails with a 429 or a 503, Scrapy retries it, and because the proxy middleware runs first, the retry leaves from a new address instead of the one that just got throttled. That combination, rotate plus retry, is what turns a fragile crawl into one that finishes.

• Keep DOWNLOAD_DELAY and AUTOTHROTTLE sane so you are not a burst even through a pool.
• Put the transient codes in RETRY_HTTP_CODES and leave 404 out, since retrying a real answer wastes exits.
• Let concurrency and rotation scale together, but do not set CONCURRENT_REQUESTS so high that you overwhelm one target.

What the pool needs

A Scrapy crawl leans hard on the address pool. WinGate fits it well: private IPv4 proxies with SOCKS5 and automatic rotation from a worldmix pool, unlimited traffic so long crawls do not meter you, and support for up to 5000 threads to match Scrapy's concurrency. The rotating endpoint cycles exits for you, so the middleware above stays a one-liner. It speaks HTTP, HTTPS, and SOCKS5, so request.meta["proxy"] works with any of them.

An honest note: middlewares and rotation stop the address from being the reason your spider dies, they do not make aggressive crawling acceptable. Keep AutoThrottle on, respect robots and the site's terms, and pace your concurrency. There is a free 2 hour test, so wire the middleware in and run a small crawl to watch the success and retry rates before you scale up.

The takeaway: put the proxy in a downloader middleware, register it before RetryMiddleware, list the transient codes to retry, and point it all at a rotating pool. Do that and Scrapy crawls at the scale it was built for without getting blocked on the first burst.