
When you build a web scraper, sooner or later you hit the same wall: your requests work locally, but fail at scale because of rate limits, IP blocks, or geo-restrictions.
A proxy layer helps you separate the scraping logic from the network access layer. In this post, I’ll show a simple Python setup for rotating residential proxies, plus a few practical tips for keeping the scraper stable.
Thordata is one option for this kind of workflow: it offers residential, mobile, static ISP, and datacenter proxies, with residential IP coverage across 190+ countries.
Why use a proxy for scraping?
Proxies are useful when you need to:
Collect publicly available data at scale
Test how a page behaves from different locations
Avoid having all requests originate from a single IP
Handle geo-specific content, pricing, or search results
Build more resilient automation workflows
For most scraping projects, I start with rotating residential proxies because they are suitable for targets that apply stricter anti-bot controls. Thordata states that its residential pool includes over 100 million real-user IPs and supports rotation as well as sticky sessions.
Python example: rotating proxy requests
Here’s a minimal example using requests. Replace the credentials and endpoint with the values from your own proxy dashboard.
python
import os
import requests
import time
import random
PROXY_HOST = os.getenv("PROXY_HOST")
PROXY_PORT = os.getenv("PROXY_PORT")
PROXY_USER = os.getenv("PROXY_USER")
PROXY_PASS = os.getenv("PROXY_PASS")
PROXY_URL = (
f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
)
proxies = {
"http": PROXY_URL,
"https": PROXY_URL,
}
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def fetch(url: str, retries: int = 3):
for attempt in range(1, retries + 1):
try:
response = requests.get(
url,
proxies=proxies,
headers=headers,
timeout=30,
)
if response.status_code == 200:
return response.text
print(f"Attempt {attempt}: HTTP {response.status_code}")
except requests.RequestException as error:
print(f"Attempt {attempt}: {error}")
time.sleep(random.uniform(1, 3))
raise RuntimeError(f"Failed to fetch {url}")
if name == "main":
html = fetch("https://example.com")
print(html[:500])
A few important details:
Store credentials in environment variables, not in your source code.
Set a realistic User-Agent and request timeout.
Add retries with randomized delays; scraping systems should degrade gracefully.
Only scrape data you are legally permitted to collect, and respect the target site’s terms, robots directives, and rate limits.
Rotating vs. sticky sessions
Rotating sessions give you a different IP for requests, which is useful for broad collection tasks. Sticky sessions keep the same IP for a period of time, which can help when a flow requires continuity.
Thordata lists both rotating and sticky session options, and its residential product page describes configurable sessions. In practice, I use rotation for independent page requests and sticky sessions when a sequence of requests needs to look like it comes from the same client.
A simple scraping checklist
Before running a scraper in production, I check:
Does the site allow this kind of access?
Am I collecting only public data?
Can I reduce request frequency without breaking the task?
Do I handle HTTP errors, timeouts, and malformed responses?
Are credentials stored securely?
Can I cache responses to avoid repeated requests?
Do I log failures without exposing proxy credentials?
A proxy improves network resilience, but it does not replace good scraping engineering. Rate limiting, caching, retries, and respectful crawling practices still matter.
Top comments (0)