DEV Community

Mary
Mary

Posted on Edited on

How I Set Up Rotating Proxies for Python Web Scraping with Thordata


When you build a web scraper, sooner or later you hit the same wall: your requests work locally, but fail at scale because of rate limits, IP blocks, or geo-restrictions.

A proxy layer helps you separate the scraping logic from the network access layer. In this post, I’ll show a simple Python setup for rotating residential proxies, plus a few practical tips for keeping the scraper stable.

Thordata is one option for this kind of workflow: it offers residential, mobile, static ISP, and datacenter proxies, with residential IP coverage across 190+ countries.

Why use a proxy for scraping?
Proxies are useful when you need to:

Collect publicly available data at scale

Test how a page behaves from different locations

Avoid having all requests originate from a single IP

Handle geo-specific content, pricing, or search results

Build more resilient automation workflows

For most scraping projects, I start with rotating residential proxies because they are suitable for targets that apply stricter anti-bot controls. Thordata states that its residential pool includes over 100 million real-user IPs and supports rotation as well as sticky sessions.
Python example: rotating proxy requests
Here’s a minimal example using requests. Replace the credentials and endpoint with the values from your own proxy dashboard.
python
import os
import requests
import time
import random

PROXY_HOST = os.getenv("PROXY_HOST")
PROXY_PORT = os.getenv("PROXY_PORT")
PROXY_USER = os.getenv("PROXY_USER")
PROXY_PASS = os.getenv("PROXY_PASS")

PROXY_URL = (
f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
)

proxies = {
"http": PROXY_URL,
"https": PROXY_URL,
}

headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}

def fetch(url: str, retries: int = 3):
for attempt in range(1, retries + 1):
try:
response = requests.get(
url,
proxies=proxies,
headers=headers,
timeout=30,
)

        if response.status_code == 200:
            return response.text

        print(f"Attempt {attempt}: HTTP {response.status_code}")

    except requests.RequestException as error:
        print(f"Attempt {attempt}: {error}")

    time.sleep(random.uniform(1, 3))

raise RuntimeError(f"Failed to fetch {url}")
Enter fullscreen mode Exit fullscreen mode

if name == "main":
html = fetch("https://example.com")
print(html[:500])
A few important details:

Store credentials in environment variables, not in your source code.
Set a realistic User-Agent and request timeout.
Add retries with randomized delays; scraping systems should degrade gracefully.
Only scrape data you are legally permitted to collect, and respect the target site’s terms, robots directives, and rate limits.
Rotating vs. sticky sessions
Rotating sessions give you a different IP for requests, which is useful for broad collection tasks. Sticky sessions keep the same IP for a period of time, which can help when a flow requires continuity.

Thordata lists both rotating and sticky session options, and its residential product page describes configurable sessions. In practice, I use rotation for independent page requests and sticky sessions when a sequence of requests needs to look like it comes from the same client.
A simple scraping checklist
Before running a scraper in production, I check:

Does the site allow this kind of access?

Am I collecting only public data?

Can I reduce request frequency without breaking the task?

Do I handle HTTP errors, timeouts, and malformed responses?

Are credentials stored securely?

Can I cache responses to avoid repeated requests?

Do I log failures without exposing proxy credentials?

A proxy improves network resilience, but it does not replace good scraping engineering. Rate limiting, caching, retries, and respectful crawling practices still matter.

Top comments (0)