Rotating vs Sticky Residential Proxies in Python (requests, threads & Scrapy)

If you've scraped anything at scale, you've hit the wall: the first 50 requests work, then you get 403s, CAPTCHAs, or empty pages. The site has noticed that one IP is asking for a lot.

Residential proxies fix this by routing each request through a real household connection, so your traffic looks like many ordinary visitors instead of one busy server. But how you rotate those IPs matters as much as having them. This post covers the two modes you'll actually use, rotating and sticky, with Python you can paste and run.

I'll use Zyrox Proxy for the examples, but the patterns apply to most residential providers that take options in the username.

The connection string

Everything goes through one endpoint:

http://USERNAME:[e-mail]:7000

Options ride in the username as -key-value pairs. No API calls, no port juggling:

USERNAME-country-us-session-abc123-sessiontime-60

Option	What it does
country-us	Exit from a US IP (ISO country code, lowercase)
session-abc123	Keep the same IP for every request using that name
sessiontime-60	Hold it for 60 minutes (default 30, max 120)

Mode 1: Rotating (a new IP every request)

No session = rotating. This is the default, and it's what you want when each request stands alone: crawling product pages, sampling search results, checking prices across many URLs.

import os
import requests

USER = os.environ["ZYROX_USER"]
PASS = os.environ["ZYROX_PASS"]

proxy = f"http://{USER}-country-us:{PASS}@host.zyroxproxy.com:7000"
proxies = {"http": proxy, "https": proxy}

for _ in range(3):
    r = requests.get("https://api.ipify.org", proxies=proxies, timeout=30)
    print(r.text)

Run it and you'll see three different US IPs.

Note: the scheme is http:// even though the target is https://. That describes how your client talks to the proxy. HTTPS traffic is tunnelled with CONNECT and stays end-to-end encrypted.

Mode 2: Sticky (hold one IP)

Some workflows break if the IP changes mid-way: logging in and then fetching an account page, stepping through a multi-page checkout, or paginating a search where the site ties results to your session.

Add a session name and every request carrying that name exits from the same IP:

proxy = f"http://{USER}-country-us-session-checkout1:{PASS}@host.zyroxproxy.com:7000"
proxies = {"http": proxy, "https": proxy}

with requests.Session() as s:
    s.proxies.update(proxies)
    print(s.get("https://api.ipify.org", timeout=30).text)
    print(s.get("https://api.ipify.org", timeout=30).text)  # same IP

The name is just a label you choose. Change it and you get a fresh IP.

One honest caveat: a sticky IP is held, not frozen. Residential IPs belong to real devices, and if that device goes offline your session moves to a new IP. Write code that tolerates it (more on retries below).

A tiny helper so you never hand-build URLs again

def proxy_url(**opts: str) -> str:
    username = USER + "".join(f"-{k}-{v}" for k, v in opts.items())
    return f"http://{username}:{PASS}@host.zyroxproxy.com:7000"

proxy_url()                                   # rotating, any country
proxy_url(country="de")                       # rotating, Germany
proxy_url(country="gb", session="u42", sessiontime="90")  # sticky UK IP for 90 min

Many sticky sessions in parallel

Need to look like 20 independent visitors at once? Use 20 session names. There's nothing to allocate in advance:

from concurrent.futures import ThreadPoolExecutor

def fetch(worker_id: int) -> str:
    p = proxy_url(country="us", session=f"worker{worker_id}")
    r = requests.get("https://api.ipify.org",
                     proxies={"http": p, "https": p}, timeout=30)
    return f"worker {worker_id}: {r.text}"

with ThreadPoolExecutor(max_workers=5) as pool:
    for line in pool.map(fetch, range(1, 6)):
        print(line)

Five workers, five IPs, and each worker keeps its own IP across its requests.

Scrapy: proxy every request with one middleware

Setting meta["proxy"] by hand gets old fast. A small downloader middleware applies it crawl-wide, and lets a spider opt into a sticky session per request chain:

# middlewares.py
import os

class ZyroxProxyMiddleware:
    def __init__(self, country):
        self.user = os.environ["ZYROX_USER"]
        self.password = os.environ["ZYROX_PASS"]
        self.country = country

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.get("ZYROX_COUNTRY", "us"))

    def process_request(self, request, spider):
        username = f"{self.user}-country-{self.country}"
        if session := request.meta.get("zyrox_session"):
            username += f"-session-{session}"
        request.meta["proxy"] = (
            f"http://{username}:{self.password}@host.zyroxproxy.com:7000"
        )

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ZyroxProxyMiddleware": 749,  # must run before 750
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}

Then in a spider: yield scrapy.Request(url, meta={"zyrox_session": "login1"}) for anything that has to stay on one IP, and plain requests for everything else.

Retries that expect IP changes

With rotating proxies, a failed request is cheap to retry: the next attempt gets a new IP automatically.

import time

def get_with_retry(url, attempts=3, **opts):
    for i in range(attempts):
        p = proxy_url(**opts)
        try:
            r = requests.get(url, proxies={"http": p, "https": p}, timeout=30)
            if r.status_code < 500 and r.status_code != 429:
                return r
        except requests.RequestException:
            pass
        time.sleep(2 ** i)  # back off: 1s, 2s, 4s
    raise RuntimeError(f"failed after {attempts} attempts: {url}")

Which one should you use?

You need	Use
Independent requests, maximum IP diversity	Rotating (no session)
A login that survives the next request	Sticky
N different "visitors" at the same time	N session names
Results for a specific market	Add country-xx to either

A few things that will save you debugging time

• Quote proxy URLs in your shell. Characters like & in passwords get mangled otherwise.
• With curl over SOCKS5, use socks5h://, not socks5://, so DNS resolves at the exit IP. Otherwise geo-aware sites may answer for your location.
• Always set a timeout. A residential hop adds latency, and requests waits forever by default.
• Respect the sites you collect from. Public data only, sensible request rates, and honour robots.txt where it applies.

Full code

Every example above, plus Node.js, Go, PHP, Java, Playwright, Puppeteer and Selenium versions, is in a runnable repo:

👉 github.com/zyroxproxy/proxy-examples

More setup guides are in the Zyrox docs, and if you want to try this yourself, plans are pay-as-you-go from $2/GB with a balance that never expires.

What's your rotation strategy? Per request, per session, or something smarter? Tell me in the comments.