Rotating IP addresses has a reputation problem. Everyone agrees you need it, and everyone also quietly suspects it does not work that well. The complaints sound the same: "we got banned anyway," "every IP was bad," "it was slower than going direct." Some of that is bad provider quality. Most of it is not.
The part that decides whether rotation works is the pattern sitting on top of the IPs. A flawless IP pool hammered with twenty requests per second from a known datacenter ASN gets blocked in minutes. A mediocre residential IP that makes three requests with realistic pauses between them sails through. The IP list matters. The behavior matters more.
This article covers five strategies that address the behavior side. None of them are exotic. They are the things most setups skip because they feel obvious or slow, and skipping them is exactly what gets you blocked.
Strategy 1: Throttle Before You Rotate
The most common mistake is the proxy choice itself. The request rate is the real problem. Developers rotate proxies and then still send ten or fifteen requests per second, which is the single most legible bot signal there is. Rotation does not fix a request rate that no human would ever produce.
Add delays between requests and vary them. For moderate targets, two to five seconds between requests works. For sensitive targets like logins or financial data, five to ten seconds or more. The variation matters as much as the delay itself. A fixed two-second pause is a pattern. A random two-to-five-second pause is behavior.
import random
import time
import requests
def request_with_delay(url, proxies):
delay = random.uniform(2, 5) # Random 2-5 second delay
time.sleep(delay)
return requests.get(url, proxies=proxies)
# Usage: space your calls. Do not batch them.
proxies = {"http": "http://user:pass@gateway:7000", "https": "http://user:pass@gateway:7000"}
for url in target_urls:
resp = request_with_delay(url, proxies)
if resp.status_code == 200:
process(resp)
This slows you down. It also lets you keep going. A scraper that respects rate limits can run for weeks. A scraper that does not gets banned in hours, and no amount of IP rotation saves it.
Strategy 2: Rotate by Session, Not by Timer
Rotating proxies every N seconds regardless of what you have done with them is the second most common mistake. It wastes your pool and creates a detectable rhythm. A better approach ties rotation to actual usage.
Session-based rotation means each IP handles a small batch of requests, then retires. Five to ten requests per IP is a reasonable default for moderate workloads. The IP gets used, proves itself trustworthy with a few normal-looking requests, and then steps aside. This is what a real user's traffic looks like: a burst of activity, then a pause while the user reads or clicks elsewhere.
import itertools
import requests
class SessionRotator:
def __init__(self, proxy_pool, requests_per_session=10):
self.pool = itertools.cycle(proxy_pool)
self.current = next(self.pool)
self.count = 0
self.requests_per_session = requests_per_session
def get_proxy(self):
if self.count >= self.requests_per_session:
self.current = next(self.pool)
self.count = 0
self.count += 1
return self.current
# Each IP handles up to 10 requests, then rotates.
# Adjust requests_per_session based on target sensitivity.
rotator = SessionRotator(proxy_pool, requests_per_session=10)
for url in target_urls:
proxy = rotator.get_proxy()
resp = requests.get(url, proxies={"http": proxy, "https": proxy})
For light monitoring, time-based rotation with a small pool is fine. For heavy scraping at hundreds of requests per minute, per-request rotation with a large pool is the only option that avoids clustering. The middle ground, session-based rotation with twenty to fifty IPs, covers most real workloads.
Strategy 3: Match Proxy Type to Target Sensitivity
Datacenter proxies are fast, cheap, and instantly recognizable. They come from cloud provider IP blocks, and any anti-bot system that checks ASN ownership can spot them immediately. They work fine for open APIs, public pages, and targets with weak bot detection.
Residential proxies route through real ISP connections. They cost five to ten times more but look like actual home users. They are worth the premium when the target has real anti-bot infrastructure, which is most sites that hold valuable data.
The hybrid approach is what most setups should run. Start with datacenter proxies for the bulk of your traffic. Keep a smaller residential pool for the targets that actually need it. Most scraping jobs fall into the first category. The sites that fight back are the exception, not the rule.
| Proxy Type | Cost per GB | Detection Risk | Best For |
|---|---|---|---|
| Datacenter | $0.50–$2 | High | Open APIs, public pages |
| Residential | $5–$30 | Low | Login flows, guarded sites |
| Mobile | $15–$50+ | Very low | App-native, heavily fingerprinted |
One of the providers that builds its plans around this distinction is 2Extract, which bills residential and mobile proxies per port rather than per GB on some plans, on the theory that a bandwidth-heavy scraping job and a bursty session-based automation job should not be priced the same way. The provider question is worth revisiting once you know which workload you are actually running.
Strategy 4: Plan for Bans, Because They Happen
Even with good rotation and sensible throttling, bans happen. The difference between losing an hour of work and losing a day is what you do when a request comes back with a 403 or a captcha.
Rotate your request headers alongside your IP. A fresh IP carrying a stale user-agent or no referer is more suspicious than a reused IP with consistent headers. Keep a small pool of realistic user-agents and rotate them independently of the proxy pool.
import random
import requests
USER_AGENTS = [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0",
]
def request_with_headers(url, proxy):
headers = {
"User-Agent": random.choice(USER_AGENTS),
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://www.google.com/",
}
return requests.get(url, proxies={"http": proxy, "https": proxy}, headers=headers)
Track your success rates per IP. An IP that accumulates errors is burning money whether you know it or not. Retire it. Most bans come from predictable patterns, not from bad IPs, so logging and retiring is more effective than buying a bigger pool.
Strategy 5: Measure What Actually Matters
The proxy that works is rarely the one with the most IPs or the cheapest GB rate. It is the one whose sessions survive, whose IPs are distributed across enough subnets and ASNs to look like real users, and whose pricing matches your actual usage pattern.
Ask providers for session survival rate before you ask for IP count. A hundred IPs from one narrow subnet behaves like one crowded neighborhood to a detection system, not like a hundred independent identities. Ask what percentage of sessions complete without the IP dropping mid-task. That number predicts real-world reliability better than pool size.
What This Looks Like in Practice
Putting it together, a resilient setup has five layers:
- Delays. Randomized pauses that keep request rates in human territory.
- Session-based rotation. Each IP handles a small batch, then retires.
- Mixed proxy types. Datacenter for volume, residential for sensitive targets.
- Header rotation. Fresh user-agents and referers alongside fresh IPs.
- Monitoring. Track success rates, retire failing IPs, adjust pacing.
None of these layers is expensive. The expensive part is buying the wrong proxy type for the workload, which is what most people do because they guess instead of measure.
The Short Version
Proxy rotation is not a procurement problem. You do not need the most IPs or the cheapest rate. You need traffic that looks like it came from a person who is doing something specific and then pausing. Get the pacing right, rotate by session rather than by timer, match the proxy type to the target's sensitivity, and measure what breaks. The IPs are the easy part.
Top comments (3)
From the detection side, the subnet point is the one that actually gets scored. We bucket a /24 by the share of neighbors already flagged: zero is clean, under 10% tolerable, past 30% and every address in it is dirty. ASN firstSeen/lastSeen catches the rest, since "static residential" ranges that only recently left datacenter allocation still read as datacenter.
The /24 neighbor-share model reframes the provider question in a Different way. "how many IPs do you have?" is the wrong question, cus what matters is "what share of your /24s are already flagged?"
A pool of a thousand IPs concentrated in twenty /24s where 50% of the neighbors are dirty is functionally worse than a pool of fifty IPs spread across clean ranges
Exactly. Whole cloud ranges get mislabeled too, so datacenter hits need two independent sources.