Why Rotating Proxies Alone Stopped Working for My Scraper

Chris — Thu, 10 Sep 2026 02:43:59 +0000

I spent the better part of last month debugging a scraper that had been running fine for almost a year. The symptom was familiar: requests started returning 403s, then CAPTCHAs, then nothing at all. I did what I always did. I added more IPs to the pool. I shortened the rotation interval. I swapped out the proxy provider entirely.

It didn't help.

That's when I realized the thing doing the blocking had changed what it was looking at. And my architecture was built for a problem that no longer existed.

The Old Model: IP Address Was the Signal

For years, the mental model was simple. A website sees a request coming from an IP address. If that IP has made too many requests too fast, or if the IP belongs to a known datacenter range, it gets blocked. The fix was equally simple: rotate IPs. Use residential proxies, spread requests across thousands of addresses, and the blocking system loses track of you.

This model still exists in places. Sites with basic rate limiting, simple IP reputation checks, or no behavioral analysis at all. If your target falls into this category, rotation works perfectly fine.

But the sites that matter — the ones with real data, the ones behind Cloudflare, Akamai, DataDome, or HUMAN — are running something different now.

What Actually Gets Checked Today

Modern anti-bot systems don't judge a single request in isolation. They evaluate the pattern of behavior across an entire session. A few signals show up repeatedly across these platforms:

Session continuity. A real visitor arrives, looks around, and takes a sequence of actions that build on each other. Cookies persist. A session token carries across pages. The same client keeps showing up in a way that looks like one visitor rather than a new stranger every time.

Browser environment consistency. A genuine browser has a stable, internally consistent set of characteristics — rendering behavior, available APIs, hardware-reported details — that stay the same request to request. A scraping setup that swaps configuration on every attempt produces a client that looks like a different device every single request. That inconsistency is itself an anomaly.

Timing and pacing. Human interaction has natural variability. A script firing requests at a fixed interval, or firing them faster than a person plausibly could, stands out against that baseline.

The systems combine these signals into something closer to a running confidence score than a single pass/fail check.

The Problem With My Architecture

The pattern I was using — one request per IP, rotate on every attempt, discard any notion of session — was built for a world where the IP address was the primary signal. Against a behavioral system, that exact pattern is what stands out.

A client that shows up once, has no session history, presents a slightly different environment fingerprint than the last "visitor," and repeats this every few seconds looks less like a large number of different humans and more like exactly what it is: one automated process cycling through addresses. The architecture built to solve the old problem actively produces the signal the new problem is designed to catch.

What I Changed

The shift wasn't about chasing every new fingerprinting technique. That's a losing game. It was about rethinking the shape of the architecture around a few principles.

Treat sessions, not individual requests, as the unit of work. Instead of a new identity for every request, group related requests into a session that persists for a realistic span, carrying its own cookies and state the way a genuine browsing session would.

Here's what that looks like in practice. Instead of this:

# What I used to do: new IP every request
for url in urls:
    proxy = get_random_proxy()
    response = requests.get(url, proxies={"https": proxy})

I switched to a session-based approach:

import requests

session = requests.Session()
session.proxies = {
    "https": "http://user-session-abc123:pass@gate.provider.com:10001"
}

# All requests share the same IP for the session duration
for url in urls:
    response = session.get(url)

The key detail is the session-abc123 parameter in the proxy username. That tells the proxy provider to keep the same exit IP for all requests carrying that session identifier. Sticky sessions like this typically last anywhere from 1 to 30 minutes, depending on the provider and plan.

Keep the client environment internally consistent for the life of a session. Whatever combination of browser engine, headers, and configuration a session starts with should stay stable for as long as that session lasts. Consistency itself is part of what a legitimate visitor looks like.

Use rotation strategically, not reflexively. Rotation still has a role — but it should be a deliberate decision tied to identity change, not a background behavior that fires on every request. If a session is blocked, rotate. If it's working, let it run.

A Diagnostic Worth Building

One pattern that has helped me repeatedly: when a request fails, find out what actually got blocked before you rotate anything. A simple diagnose() function that runs before any remediation logic and logs whether the failure was IP-scoped, session-scoped, or something else. Only the IP-scoped branch is allowed to call rotate().

That single pattern would have saved me about two weeks.

The Takeaway

If your scraper is getting blocked and your first instinct is still "add more IPs," it's worth asking when that fix last actually worked cleanly. The honest answer for a lot of teams is "a while ago." That doesn't mean proxies are useless — good residential and ISP proxies are still a base requirement. It means the architecture around them has to account for what the blocking system is actually evaluating.

Sessions, not requests. Consistency, not randomness. Diagnose before you rotate.

If you're working on something similar and want to compare notes, I'd love to hear what's working for you.