Web scraping in 2026 is a cat-and-mouse game. Sites detect datacenter IPs in milliseconds, rate-limit by fingerprint, and serve CAPTCHAs before your parser even runs. If your scraper works perfectly at 10 requests and dies at 1,000, the problem is almost never your code — it's your IP strategy.
This guide walks through the full stack: why residential proxies matter, how rotation and sticky sessions actually work, and complete, runnable Python code for each pattern.
Why Datacenter IPs Fail First
Anti-bot systems (Cloudflare, PerimeterX, Akamai, DataDome) score every request on multiple signals. The fastest one to check is the IP itself:
Datacenter IPs belong to AWS, Hetzner, OVH and other hosting providers. Real humans don't browse from server racks, so these IPs start with a high risk score before any other signal is evaluated.
Residential IPs belong to real ISPs and real devices. To the target site, your request looks like a normal user in Berlin or Austin browsing from home.
That's the entire value proposition: residential IPs buy you a neutral starting score instead of a guilty one. Your headers, cookies, and request patterns still matter — but they matter after the IP check.
Setup
You'll need Python 3.9+ and two packages:
pip install requests beautifulsoup4
For a proxy provider, I'll use Thordata in the examples below — their residential pool covers 100M+ IPs across 190+ countries, and new users can grab up to 500MB of free trial traffic, which is enough to run every example in this guide (coupon code thor020 gets you a discount if you continue). The patterns work with any provider that exposes a username/password gateway, so swap in your own credentials if you prefer.
Grab your credentials from the provider's dashboard. Every example below assumes two variables:
PROXY_HOST = "gate.thordata.com" # replace with your gateway host
PROXY_PORT = 9000 # replace with your gateway port
PROXY_USER = "your-username"
PROXY_PASS = "your-password"
Pattern 1: Basic Single Request Through a Residential Proxy
Start simple. One request, one residential IP:
import requests
def get_proxy():
return {
"http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
"https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}
response = requests.get(
"https://httpbin.org/ip",
proxies=get_proxy(),
timeout=30,
)
print(response.json())
{"origin": "85.x.x.x"} <- a residential IP, not your real one
Run this and you'll see an IP that isn't yours. You've just made your first unflagged request.
Pattern 2: IP Rotation on Every Request
Rotation is the default behavior of most residential gateways: each new connection gets a different IP from the pool. This matters when a site limits requests per IP:
import requests
from itertools import cycle
URLS = [f"https://httpbin.org/ip?i={i}" for i in range(5)]
for url in URLS:
try:
r = requests.get(url, proxies=get_proxy(), timeout=30)
print(r.json()["origin"])
except requests.RequestException as e:
print(f"failed: {e}")
Each printed IP should be different. If a site allows 50 requests per IP per hour, rotating across the pool effectively removes that ceiling — but don't rotate blindly for everything. Some sites expect session consistency.
Pattern 3: Sticky Sessions for Login Flows
Here's the mistake everyone makes once: they rotate IPs on every request, log in successfully, and then get kicked out on the next request because "the user's IP changed mid-session." That's suspicious behavior no real human produces.
Sticky (session) proxies keep the same IP for a defined window — typically up to 30 minutes — which is what you need for login flows, multi-page checkouts, and paginated dashboards. With most gateways including Thordata, you activate it by appending a session ID to the username:
import requests
import uuid
def sticky_proxy(session_id: str, geo: str = "us"):
user = f"{PROXY_USER}-session-{session_id}-country-{geo}"
return {
"http": f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
"https": f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}
One session = one consistent residential IP
session_id = uuid.uuid4().hex[:12]
for page in range(1, 4):
r = requests.get(
f"https://httpbin.org/ip?page={page}",
proxies=sticky_proxy(session_id, geo="us"),
timeout=30,
)
print(f"page {page}: {r.json()['origin']}")
All three pages: the SAME IP — the site sees one consistent "user"
Note the -country-us fragment — that's geo-targeting, covered next.
Pattern 4: Geo-Targeting
Search results, prices, and ads change by location. A request from a German IP and a US IP to the same URL returns different content. If you're building a price tracker or SERP monitor, you don't just need an IP — you need an IP in the right country:
for country in ["us", "de", "jp", "br"]:
user = f"{PROXY_USER}-country-{country}"
proxy = {
"http": f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
"https": f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}
r = requests.get("https://httpbin.org/ip", proxies=proxy, timeout=30)
print(country, "->", r.json()["origin"])
Thordata supports country, city, and ASN-level targeting — the last one matters when you want to test what users of a specific ISP (say, Comcast or Deutsche Telekom) see.
Pattern 5: A Real Scraper with Rotation, Retries, and Throttling
Putting it together — a small, production-shaped scraper:
import time
import random
import requests
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def fetch(url: str, geo: str = "us", max_retries: int = 4) -> str | None:
user = f"{PROXY_USER}-country-{geo}"
proxy = {
"http": f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
"https": f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}
for attempt in range(max_retries):
try:
r = requests.get(url, headers=HEADERS, proxies=proxy, timeout=30)
if r.status_code == 200:
return r.text
if r.status_code in (403, 429):
# Blocked or rate-limited: back off, retry with a new IP
time.sleep(2 ** attempt + random.random())
continue
return None
except requests.RequestException:
time.sleep(2 ** attempt)
return None
def scrape(urls: list[str], geo: str = "us"):
for url in urls:
html = fetch(url, geo=geo)
if html:
print(f"OK {url} ({len(html)} bytes)")
else:
print(f"FAIL {url}")
# Human-ish pacing: fast enough to work, slow enough to look real
time.sleep(random.uniform(1.5, 4.0))
scrape([
"https://example.com/page1",
"https://example.com/page2",
])
Three things make this survive in production:
Exponential backoff with jitter — a 429 means "slow down," not "try harder."
Per-request rotation — every retry lands on a fresh residential IP.
Randomized delays — perfectly uniform timing is itself a bot signal.
Rotating vs Sticky: The Decision Table
Scenario Use
One-off page fetches Rotating
Mass crawling public pages Rotating + throttling
Login → browse → paginated data Sticky session
Checkout / booking flows Sticky session
SERP tracking per country Rotating + geo-targeting
Testing what one ISP's users see ASN-targeted sticky
Scaling Beyond requests
For serious throughput, graduate from requests to an async stack:
import asyncio
import aiohttp
async def fetch(session, url, proxy_url):
async with session.get(url, proxy=proxy_url, timeout=aiohttp.ClientTimeout(total=30)) as r:
return await r.text()
async def main(urls):
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
async with aiohttp.ClientSession() as session:
results = await asyncio.gather(*(fetch(session, u, proxy_url) for u in urls))
for r in results:
print(len(r), "bytes")
asyncio.run(main([f"https://httpbin.org/ip?i={i}" for i in range(10)]))
Async gets you concurrency; the residential pool gets you survival at scale. You need both.
Wrap-Up
The mental model that makes proxy strategy click: rotation is for when the site counts requests per IP, stickiness is for when the site remembers who you are, and geo-targeting is for when the site serves different content by location. Get those three decisions right and 90% of blocking problems disappear.
Every pattern in this guide runs on a free trial: Thordata gives new users up to 500MB of test traffic (100M+ residential IPs, 190+ countries, city and ASN targeting) — sign up
here
and use code thor020 for a discount when you're ready to scale.
This article is part of a series on production web scraping. Questions or corrections? Drop a comment.
Top comments (0)