Most scraping tutorials treat proxies as a one-liner: plug in a URL, set proxies={"https": ...}, done. Then the blocks start, and people blame the proxy provider. Nine times out of ten, the provider isn't the problem — the pattern is. Here are three patterns that changed my results.
Pattern 1: Rotate per request, but pin the geography.
Random rotation across a global pool sounds safe but hurts you twice: latency, and region-mismatched content. Pin each request to the country (or city) of the target, and rotate within that region:
import requests
def fetch(url, country):
proxy = f"http://user-cc-{country}:pass@gw.thordata.com:8080"
return requests.get(url, proxies={"http": proxy, "https": proxy},
timeout=15)
Region-pinned rotation means every response is what a local visitor sees — which matters for pricing pages, search results, and anything geo-gated.
Pattern 2: Sticky sessions for logged-in flows.
Rotating mid-session breaks cookies and triggers re-authentication — a classic ban signal. For anything behind a login, hold one IP for the session's lifetime. Most residential providers support this; Thordata, for example, holds a sticky IP for up to 90 minutes via a session ID in the proxy string. Rule of thumb: anonymous crawl = rotate; authenticated flow = stick.
Pattern 3: Think in cost-per-page, not cost-per-IP.
Per-IP pricing hides the real math. At $0.65/GB, a 300KB page costs about 0.02 cents. Ten thousand pages ≈ $1.95. Run that against whatever you're paying for "dedicated IPs" — the per-GB model usually wins once you stop renting dead IPs.
The unifying idea: a proxy isn't an anonymizer, it's a routing layer — and routing decisions (region, session, cost) belong in your code, not in a dashboard somewhere.
I collect web data for a living and currently work with Thordata's residential network (100M+ IPs, free trial here:
thordata.com
) — but these patterns apply to any provider with geo-targeting and session control.
Top comments (0)