Aethyn Team

Posted on Jul 8

The Residential Proxy "Pool Size" Myth — Why 100M+ IPs Tells You Almost Nothing

#webscraping #python #proxy #datascience

Open any residential proxy provider's homepage and the first thing you see is a number. 50 million IPs. 100 million. "The largest residential network in the world." It's the industry's favourite headline — and one of the weakest predictors of whether your scraper actually works.

What the "pool size" number actually measures

That headline figure is almost always a cumulative count of unique IPs seen over a rolling 30-day window — not a live inventory. Three things follow, none of which make the marketing:

It's self-reported and unaudited. No standard for what counts as an IP, over what window. Two networks drawing on the same underlying supply can honestly print wildly different numbers.
The incentive only runs one way. Nobody ever chose a provider for a smaller pool. Every incentive pushes the number up — the exact condition under which a metric stops meaning anything.
Nobody publishes the number that matters: concurrent addresses available in your country, right now. That's a fraction of the headline and you'll never see it on a pricing page.

The headline IP count is a marketing decision, not a measurement.

Why the number can't predict success

The target never sees your pool. It sees one IP — the one you're using right now — and it scores that single address. Modern anti-bot systems judge a request on:

Is this a residential IP at all? The ASN instantly separates a real consumer ISP from a datacenter range. Datacenter IPs are blocked on sight.
What's this IP's reputation? Fraud-score databases flag addresses recently tied to abuse or automation.
Has this IP already misbehaved here? A recycled exit that hammered the target yesterday is treated very differently from a quiet one.
Is the story consistent? An IP geolocating to one country while headers/timezone claim another is a tell.

None of those is pool size. A huge pool full of recycled or flagged addresses performs worse than a smaller, clean one.

The mistake that ruins your data: trusting the status code

Internalise this before you benchmark anything: a 200 is not success. Modern defenses "soft-block" — a 200 with a CAPTCHA interstitial, a stripped decoy, or an empty shell. Measure success on content, not status:

import requests

PROXY = "http://USER:PASS@proxy.example.com:PORT"  # your provider's endpoint
proxies = {"http": PROXY, "https": PROXY}

def is_real_success(resp, positive_token):
    if resp.status_code != 200:
        return False
    body = resp.text.lower()
    block_signals = ["captcha", "unusual traffic", "access denied", "are you a robot"]
    if any(s in body for s in block_signals):
        return False
    return positive_token.lower() in body   # something ONLY a real page has

targets = [...]          # YOUR real URLs — not httpbin, not the vendor demo
TOKEN   = "add to cart"  # a string unique to a genuine page on YOUR target

ok = sum(is_real_success(requests.get(u, proxies=proxies, timeout=30), TOKEN)
         for u in targets)
print(f"content-verified success rate: {ok / len(targets):.1%}")

That percentage — success on your sites, verified on content — is the number the entire industry's marketing exists to distract you from.

Check what you're actually handed

Pull the exit IP and inspect it. This is where "big pool, dirty IPs" gets exposed in thirty seconds:

ip = requests.get("https://api.ipify.org", proxies=proxies, timeout=30).text
print("exit IP:", ip)

# Look that IP up in any public IP-reputation / fraud-score tool and confirm:
#   1. ASN is a consumer ISP (not AWS/GCP/a hosting provider)
#   2. it geolocates where you asked
#   3. it isn't already flagged for abuse

Spot-check the exits you're served — reputation beats raw count every time.

The one question a provider can't dodge

Ask where the IPs come from — and don't accept a non-answer. IPs from users who gave informed, opt-in consent behave like the real people they belong to and carry the reputation to match. IPs harvested from compromised devices are a botnet: already flagged, ethically indefensible, and a legal liability you inherit. A clean sourcing story is either documented or it isn't. A network that leads with a pool-size number and goes quiet on sourcing has told you which one it optimised for.

TL;DR — what to judge a network on

Success rate on your targets, verified on content (not status).
IP reputation — spot-check exits against a fraud-score tool.
Sourcing — consented/opt-in, not mystery pools.
Concurrency where you need it, not a global cumulative total.
Per-GB pricing you can forecast — if you need a sales call to see a price, you can't model cost.

Whatever wins that test is your provider. Pool size never enters the calculation — because it was never in the target's calculation either.

About Aethyn

I help build Aethyn — residential proxies built for engineers, not a marketing slide. One unified endpoint, country/city/session targeting encoded in the username, transparent per-GB pricing (€2.00/GB Premium, €4.50/GB Elite), and a free trial, no card. We deliberately don't lead with an IP-count — we'd rather you run the benchmark above on your own targets.

👉 Try it on your own targets: aethyn.io — free, no card

Originally published on aethyn.io.

Images via Unsplash (free license).

DEV Community