Rotating Proxies in Python: requests, httpx, and aiohttp

#python #webscraping #proxy #webdev

If you have ever pointed a Python scraper at a real target, you know the pattern. The first few hundred requests fly through, then responses slow down, then you start getting 429 and 403 status codes, and soon every request comes back as a captcha page. Your code is fine. The problem is that every request leaves from one address, and the target site counts requests per address. The fix is rotation: spread your requests across many addresses so no single one ever looks abnormal. This post shows how to wire that into the three HTTP clients most Python scrapers use.

Why one address hits a wall

A site does not need to be clever to stop a scraper. It just counts how many requests arrive from a single address inside a short window. A human browsing loads a handful of pages a minute. A scraper fires hundreds. Once you cross the threshold, the site throttles you, serves a challenge, or blocks the address outright. Slow down enough to stay under the limit and a job that should take an hour takes a week. Rotation solves this by giving each request a fresh address, so the per-address counter never climbs high enough to trip.

The requests version

The requests library takes a proxies dict per call. With a rotating endpoint you point every request at the same proxy URL and the pool hands you a different exit each time.

import requests

PROXY = "http://user:pass@proxy.host:port"
proxies = {"http": PROXY, "https": PROXY}

for url in urls:
    r = requests.get(url, proxies=proxies, timeout=20)
    print(r.status_code, url)

Add a short random delay between calls and retry a failed request on a fresh address rather than hammering the same one. That alone clears most soft blocks.

The httpx version

httpx uses almost the same shape and also gives you an async client for concurrency.

import httpx, asyncio

PROXY = "http://user:pass@proxy.host:port"

async def fetch(client, url):
    r = await client.get(url, timeout=20)
    return r.status_code

async def main(urls):
    async with httpx.AsyncClient(proxy=PROXY) as client:
        return await asyncio.gather(*(fetch(client, u) for u in urls))

Because the proxy rotates per request, concurrency and rotation work together: many requests in flight, each from a different address.

The aiohttp version

aiohttp passes the proxy per request, which fits a rotating endpoint naturally.

import aiohttp, asyncio

PROXY = "http://user:pass@proxy.host:port"

async def fetch(session, url):
    async with session.get(url, proxy=PROXY, timeout=20) as r:
        return r.status

async def main(urls):
    async with aiohttp.ClientSession() as session:
        return await asyncio.gather(*(fetch(session, u) for u in urls))

What actually matters in the proxy itself

The client code is the easy part. Whether the job survives depends on the addresses behind that endpoint.

Private addresses. Shared pools inherit the reputation of everyone who used them before you, so you get flagged before your first request. Private IPv4 answers to you alone.
Real rotation. A large pool that cycles the exit per request is what keeps each address under the rate limit. Without it, rotation is just a setting that does nothing.
Protocol coverage. HTTP, HTTPS, and SOCKS5 support means the same endpoint drops into requests, httpx, aiohttp, or anything else without a rewrite.
Unlimited traffic. Large crawls are heavy. A plan that meters every megabyte turns a scraping job into a budget decision.

This is where rotating proxies earn their place. WinGate provides private IPv4 with SOCKS5, automatic rotation from a worldmix pool, unlimited traffic, and support for up to 5000 threads, so wide concurrency runs without you throttling yourself. It speaks HTTP, HTTPS, and SOCKS5, so every snippet above works by swapping in your endpoint.

An honest note: no proxy makes aggressive scraping invisible. If you fire thousands of requests per second at one endpoint you will still draw attention, and rotation does not exempt you from a site's terms. What it buys you is room to run at a sensible pace without the address itself becoming the reason the job dies. There is a free 2 hour test, so point one of these snippets at your own targets and watch the success rate before you commit.

DEV Community