DEV Community

RoamProxy
RoamProxy

Posted on

Building a Rotating Proxy Pool in Python That Survives Failures

#automation #python #tutorial #webscraping

If you scrape the web at any real scale, a single proxy will not cut it. IPs get rate-limited, temporarily banned, or simply time out. The durable fix is a pool of proxies plus logic that rotates through them and retries failed requests on a fresh IP.

Here is how to build one in Python that does not fall over the first time a proxy misbehaves.

Why one proxy is never enough

A single IP hitting a site hundreds of times per minute looks nothing like a human. Target sites respond with 429 Too Many Requests, throw CAPTCHAs, or quietly start returning empty pages. Even a "good" residential IP has a bad day: the upstream node drops, the TCP connection hangs, DNS flakes.

So two things are non-negotiable in production:

  1. Rotation — spread requests across many IPs.
  2. Retry on a different IP — when a request fails, don't just retry the same dead proxy; pick a new one.

The naive version (and why it breaks) 

import requests

proxy = "http://user:pass@gw.roamproxy.com:41080"
r = requests.get("https://example.com", proxies={"http": proxy, "https": proxy})
Enter fullscreen mode Exit fullscreen mode

One bad IP and this raises, with no rotation and no retry. Wrap it in a loop and you're already reinventing a pool — badly. Let's do it properly.

A resilient pool

The core idea: on every attempt, pick a proxy, and treat both exceptions and retryable status codes (429, 5xx) as "try again on a fresh IP," with exponential backoff.

import random
import time
import requests


class ProxyPool:
    RETRYABLE = {429, 500, 502, 503, 504}

    def __init__(self, proxies, max_retries=3, timeout=15):
        self.proxies = list(proxies)
        self.max_retries = max_retries
        self.timeout = timeout

    def _pick(self):
        return random.choice(self.proxies)

    def get(self, url, **kwargs):
        last_exc = None
        for attempt in range(self.max_retries):
            proxy = self._pick()
            try:
                r = requests.get(
                    url,
                    proxies={"http": proxy, "https": proxy},
                    timeout=self.timeout,
                    **kwargs,
                )
                if r.status_code in self.RETRYABLE:
                    raise requests.HTTPError(f"retryable status {r.status_code}")
                return r
            except requests.RequestException as e:
                last_exc = e
                time.sleep(2 ** attempt)  # 1s, 2s, 4s...
        raise last_exc


pool = ProxyPool([
    "http://user:pass@gw.roamproxy.com:41080",
    # ...add as many gateway endpoints or credentials as you like
])

resp = pool.get("https://httpbin.org/ip")
print(resp.json())
Enter fullscreen mode Exit fullscreen mode

A few things worth calling out:

  • Rotate on every attempt, not just on success. The whole point is that attempt 2 uses a different IP than attempt 1.
  • Exponential backoff (2 ** attempt) keeps you from hammering a struggling target.
  • Treat 429/5xx as retryable, but let real client errors like 404 return normally — retrying them just wastes IPs.

With most gateway providers you don't even maintain a list of raw IPs. You point at one endpoint and get a new exit IP per request automatically. Rotation for free:

# rotating: a fresh IP on every request
proxy = "http://user-country-us:pass@gw.roamproxy.com:41080"
Enter fullscreen mode Exit fullscreen mode

When you actually want the same IP

Rotation is the default, but some flows break if the IP changes mid-way: logins, add-to-cart, anything multi-step behind a session cookie. That's a sticky session — pin a session id in the username so the gateway keeps handing you the same exit IP until you rotate it:

# sticky: same IP while the session id stays constant
sticky = "http://user-country-us-session-abc123:pass@gw.roamproxy.com:41080"

s = requests.Session()
s.proxies = {"http": sticky, "https": sticky}
s.post("https://example.com/login", data=creds)   # same IP...
s.get("https://example.com/dashboard")             # ...as this one
Enter fullscreen mode Exit fullscreen mode

Change the session id and you get a new IP. That single lever — rotate vs. pin — covers the vast majority of scraping needs.

Don't reinvent the whole thing

The pattern above (rotate, retry-on-failure with backoff, plus sticky sessions and a pool that recycles slots) is common enough that we packaged it into a tiny zero-dependency SDK: roamproxy-python. It gives you a Client, a StickySession context manager, and a SessionPool that retries on 5xx/429/connection errors out of the box:

from roamproxy import SessionPool

pool = SessionPool(retry_on_failure=True)
resp = pool.get("https://httpbin.org/ip")
Enter fullscreen mode Exit fullscreen mode

It's MIT-licensed — copy the ideas, or pip install it, whatever's faster for you.

Wrapping up

A production scraper isn't "requests + a proxy." It's a pool that rotates, retries on a fresh IP, backs off, and can pin a sticky session when a flow needs one. Build those four behaviors in and most "the site keeps blocking me" problems quietly disappear.

If you'd rather not run your own IP infrastructure, that's the problem RoamProxy solves — pay-as-you-go residential and datacenter proxies from $2/GB, with the rotating/sticky gateway shown above. Either way, the engineering above is the part that matters.

Top comments (0)