The number you track is not the number they key on
Most of us who run scrapers watch one dashboard stat for pool health: unique exit IPs per run. It feels like the right number. More distinct IPs, more diversity, less chance two of your requests look alike.
But the side doing the blocking rarely decides at the IP granularity. Abuse and fraud systems tend to correlate at a coarser level: the autonomous system, and - increasingly - the routed prefix an address lives in. A /24 that has been on a blocklist drags its neighbours with it. And a single announced prefix can be far larger than a /24. Two exit IPs whose first three octets differ completely can still sit inside one BGP advertisement, which means to a correlation engine they are the same neighborhood.
So the useful question is not "how many unique IPs did I rotate through" but "how many distinct subnets and prefixes did those IPs actually occupy, and how concentrated were they". I started measuring that on my own sessions, and the gap between the two numbers was uncomfortable enough to write down.
Two groupings, and why the cheap one lies
There are two ways to bucket an exit IP, and they answer different questions.
The cheap one is the containing /24 - just take the first three octets. It is free, offline, and a reasonable first pass, because /24 is the smallest unit most blocklists still reason about. The problem is that it silently assumes every provider announces /24s. Many do not. Carriers and large hosting providers routinely announce supernet blocks and then carve customer allocations out of them internally. The route table only knows the supernet.
The honest one is the BGP announced prefix: the actual block that shows up in the global routing table as the destination for that address. If two IPs fall under one announced /16 or /11, a correlation layer keyed on route origin sees one source, however different the addresses look.
The divergence is not a corner case. RIPEstat's looking-glass data call is free, needs no key, and will tell you the encompassing routed prefix for any address. A spot check on a few well-known public IPs - run today, 2026-09-23 - makes the point:
8.8.8.8 /24-group = 8.8.8.0/24 announced = 8.8.8.0/24 AS15169
1.1.1.1 /24-group = 1.1.1.0/24 announced = 1.1.1.0/24 AS13335
13.107.42.14 /24-group = 13.107.42.0/24 announced = 13.107.42.0/24 AS8068
151.101.0.81 /24-group = 151.101.0.0/24 announced = 151.101.0.0/22 AS54113
49.68.206.60 /24-group = 49.68.206.0/24 announced = 49.64.0.0/11 AS4134
203.0.113.5 /24-group = 203.0.113.0/24 announced = (none - not in DFZ)
Fastly's 151.101.0.81 looks like its own /24, but the route is a /22 - four /24s collapse into one advertised block. China Telecom's 49.68.206.60 collapses harder: the announced prefix is a /11, roughly two million addresses presented as one routing object. An address in 49.68.x.x and one in 49.75.x.x share that prefix, yet the /24 grouping would happily count them as fully independent. And 203.0.113.5 - a documentation range that is not actually routed - returns no prefix at all, which is itself something your harness has to handle instead of crashing.
Those are infrastructure IPs, used here only to prove the two code paths disagree in the expected direction. The real value is running the same code against your own sessions.
The audit harness
Two pieces: sample the exit IP each proxy session actually egresses as, then bucket those IPs by both /24 and announced prefix. The sampling needs your gateway; the bucketing is pure Python standard library.
Sampling one session's exit IP from the shell - the pattern to loop over your session ids:
curl -x "http://USER:PASS@gw.provider.example:9000?sessionid=123" https://icanhazip.com/
# 49.68.206.60
And to check what prefix the world actually routes that IP into:
curl -s "https://stat.ripe.net/data/looking-glass/data.json?resource=49.68.206.60" \
| grep -o '"prefix":"[^"]*"' | sort | uniq -c
The full audit, saving the requests import for the part that genuinely needs it:
# prefix_audit.py (stdlib except for the proxy sampling step)
import ipaddress, json, time, urllib.request
from collections import Counter
RIPESTAT = "https://stat.ripe.net/data/looking-glass/data.json?resource={ip}"
_cache = {}
def announced_prefix(ip):
"""(prefix, origin_asn) as seen in global BGP, via RIPEstat. Cached + fault-tolerant."""
if ip in _cache:
return _cache[ip]
prefixes, asns = Counter(), Counter()
try:
with urllib.request.urlopen(RIPESTAT.format(ip=ip), timeout=20) as r:
data = json.load(r)
for rrc in data.get("data", {}).get("rrcs", []):
for peer in rrc.get("peers", []):
if peer.get("prefix"): prefixes[peer["prefix"]] += 1
if peer.get("asn_origin"): asns[str(peer["asn_origin"])] += 1
pfx = prefixes.most_common(1)[0][0] or None # unrouted -> None
asn = asns.most_common(1)[0][0] or None
except Exception:
pfx = asn = None # rate limit / timeout -> UNKNOWN, not crash
_cache[ip] = (pfx, asn)
return pfx, asn
def sample_egress_ips(session_specs, echo="https://icanhazip.com/"):
"""Return one exit IP per proxy session spec (requests-style proxies dict)."""
import requests
ips = []
for proxies in session_specs:
for _ in range(3): # a flaky exit is not a missing prefix
try:
ips.append(requests.get(echo, proxies=proxies, timeout=15).text.strip())
break
except Exception:
time.sleep(1)
time.sleep(0.3) # polite to echo + RIPEstat
return ips
def report(ips):
n = len(ips)
by24 = Counter()
for ip in ips:
try:
by24[str(ipaddress.ip_network(ip + "/24", strict=False))] += 1
except ValueError:
pass
# dedupe at /24 level first so we spend RIPEstat calls on distinct subnets only
by_pfx, by_asn = Counter(), Counter()
for rep in by24:
pfx, asn = announced_prefix(rep)
by_pfx[pfx or "UNKNOWN/unrouted"] += 1
if asn:
by_asn[asn] += 1
print(f"sampled sessions : {n}")
print(f"unique exit IPs : {len(set(ips))}")
print(f"unique /24 (local) : {len(by24)}")
print(f"unique announced prefix : {len(by_pfx)}")
print("top /24 concentration : " + ", ".join(f"{p}={c}" for p, c in by24.most_common(3)))
print("top ASN concentration : " + ", ".join(f"AS{a}={c}" for a, c in by_asn.most_common(3)))
shares = [c / n for _, c in by24.items()]
print(f"P(two sessions share a /24) ~ {sum(s*s for s in shares):.3f}")
if __name__ == "__main__":
import sys
if len(sys.argv) > 1 and sys.argv[1] == "--demo":
report(["8.8.8.8", "1.1.1.1", "151.101.0.81",
"151.101.4.81", "49.68.206.60", "49.70.1.1"])
else:
report(sample_egress_ips(build_my_sessions())) # you supply this
build_my_sessions() is where your provider's session model plugs in: for a sticky-session gateway you emit one proxies dict per session id and loop over enough of them to sample the pool; for a rotating gateway you fire many single-shot requests and collect the distinct exits. The rest of the code does not care which.
What to watch for, and where this is wrong
The metric that actually predicts trouble is the ratio unique /24 : sampled sessions and the concentration of the top prefix, not the raw unique-IP count. A pool of 3,000 exits that lands in 40 /24s spread over 6 prefixes is behaving like a much smaller pool. If two concurrent workers are more likely than you think to share a subnet, then the subnet-level blocklist, the fraud-review queue, and the same support agent's session cache are all doing the same correlation you just measured for free.
Be honest about the limits, because they will bite in a report:
- The announced prefix is the routing granularity, not the provider's internal allocation block. A /11 route may be carved into hundreds of customer /22s. So BGP-level collapse is an upper bound on coarseness - "correlates at least this much", not "physically one data centre". The /24 grouping is the lower bound. Reality usually sits between them, and having both bounds is more useful than a single number you cannot interpret.
- RIPEstat is fair-use rate-limited (roughly one call per second). That is why the code dedupes to distinct /24s before querying, caches results, and treats any timeout as
UNKNOWN/unroutedrather than letting one dropped call crash a whole run. - A sampled exit that is momentarily unrouted or behind a anycast boundary will bucket as
UNKNOWN. Do not silently drop those; they inflate your apparent diversity, so count them. - Sample size matters. Two sessions that happen to look diverse tell you nothing about the tail. I run at least a few hundred sessions before trusting the concentration ratio, and I log the sample count next to the result so a future me knows how much to trust a given number.
None of this replaces watching real block-and-retry rates on your actual targets - it tells you why a pool that looks huge in IP-count terms can underperform. If you are choosing or scaling a residential pool on a per-GB basis (Thordata's residential line is from $0.65/GB as listed on 2026-09-23, down from a $1.05/GB list price - check the live pricing page before you budget), measure its prefix spread before you assume gigabytes scale linearly with usable diversity.
Disclosure: I work with Thordata on content. The measurement above is generic - RIPEstat, icanhazip, and a /24 vs announced-prefix comparison - and runs against any proxy provider, including yours.
Top comments (0)