Measure Proxy Pool Concentration Instead of Counting IPs

I work with 98IP, so I have a commercial connection to a proxy provider. This post is a vendor-neutral measurement method: it does not rank providers, and the example code works on a sanitized sample from any authorized proxy trial.

A rotating proxy test often ends with a triumphant number: “we saw 8,412 unique IPs.” That number is incomplete.

If 70% of successful sessions came from one ASN, the operational pool is more concentrated than the unique-IP count suggests. If your backup provider produces the same dominant networks, it may not be an independent fallback. And if sticky sessions and rotating sessions are mixed, the uniqueness metric is not even measuring one product behavior.

The useful unit is an event-level sample:

provider, region, time_block, session_mode, exit_ip,
asn, prefix_bucket, transport_ok, application_ok, latency_ms

Keep proxy credentials, cookies, authorization headers, and target payloads out of this file. Limit retention of raw IP addresses; hashed IPs can still be linkable.

Sample in blocks

Run the same blocks for each provider:

• one approved region at a time;
• several fixed windows across at least two days;
• the same request count, concurrency, pacing, and payload;
• a controlled endpoint plus an approved target class;
• rotating and sticky modes reported separately.

Do not rotate as fast as possible to enumerate a provider's inventory. You want the supply your real workload receives under normal authorized use.

Enrich carefully

RDAP provides structured registration information for an IP network. Routing data provides origin-network context. Neither proves that an address is residential, ethically sourced, exclusive, or clean.

Keep these fields distinct:

• rdap_prefix: the network object returned by the registry;
• asn: the observed route-origin grouping;
• prefix_bucket: a fixed analytical grouping, such as an IPv4 /24;
• lookup_at: when the enrichment was performed.

An analysis bucket is not an ownership boundary. Label it honestly.

Calculate concentration

Here is a compact Python example for a sanitized CSV that already contains asn and prefix_bucket:

from collections import Counter
from csv import DictReader


def concentration(values):
    counts = Counter(v for v in values if v)
    total = sum(counts.values())
    if not total:
        return {
            "observations": 0,
            "groups": 0,
            "top_share": None,
            "hhi": None,
            "effective_groups": None,
        }

    shares = [count / total for count in counts.values()]
    hhi = sum(p * p for p in shares)
    return {
        "observations": total,
        "groups": len(counts),
        "top_share": max(shares),
        "hhi": hhi,
        "effective_groups": 1 / hhi,
    }


def jaccard(left, right):
    left, right = set(left), set(right)
    union = left | right
    return len(left & right) / len(union) if union else None


with open("proxy_sample.csv", newline="") as handle:
    rows = list(DictReader(handle))

successful = [row for row in rows if row["transport_ok"] == "true"]

print("ASN", concentration(row["asn"] for row in successful))
print(
    "PREFIX",
    concentration(row["prefix_bucket"] for row in successful),
)

The Herfindahl-Hirschman Index is sum(p²) across group shares. 1 / HHI is the effective number of equally sized groups.

Suppose a sample contains 12 ASNs. If one ASN carries most sessions, the effective count may be closer to three than twelve. That is the point: observed group count says what appeared; effective count says how evenly traffic was distributed.

Do not use a universal HHI threshold. Compare candidates under the same region and workload, then set acceptance criteria based on continuity needs.

Compare backup independence

For two providers, compute Jaccard overlap for ASN sets and prefix-bucket sets:

provider_a = [r for r in successful if r["provider"] == "A"]
provider_b = [r for r in successful if r["provider"] == "B"]

asn_overlap = jaccard(
    (r["asn"] for r in provider_a),
    (r["asn"] for r in provider_b),
)

prefix_overlap = jaccard(
    (r["prefix_bucket"] for r in provider_a),
    (r["prefix_bucket"] for r in provider_b),
)

print({"asn_jaccard": asn_overlap, "prefix_jaccard": prefix_overlap})

Set overlap is only the first view. Add traffic-weighted overlap because a few shared ASNs may carry most sessions for both providers.

High overlap does not prove reselling. It shows that the tested products may not give you independent network diversity. Contractual supply-chain claims require vendor documentation.

Report by block, not only globally

A global result can hide the market you actually need. Produce one row per provider, region, time block, and session mode:

successful_sessions
unique_exit_rate
top_asn_share
top_prefix_share
asn_hhi
effective_asns
prefix_hhi
effective_prefixes
application_success_rate
p50_latency
p95_latency

Then compare repeated blocks. A one-hour sample can be dominated by a temporary supply shift. A multi-day design shows whether concentration is persistent.

What not to conclude

• A consumer ISP ASN does not prove informed-consent residential sourcing.
• More unique IPs do not compensate for poor target success.
• A repeated IP is not a defect in a sticky session.
• Route authorization does not measure latency or IP reputation.
• Low overlap in one country does not prove global independence.

Procurement output

The final deliverable should state:

1. whether critical regions meet success and sample requirements;
2. whether any ASN or prefix dominates without an acceptable explanation;
3. whether results remain stable across time blocks;
4. whether a backup provider reduces weighted network overlap;
5. whether consent, sourcing, replacement, and incident processes pass review;
6. what one successful useful result costs after retries and failures.

The full conceptual checklist is also available from 98IP at https://en.98ip.com/?k=dev.

Use this only for authorized endpoints and workloads. Respect provider terms, target rules, rate limits, privacy requirements, and applicable law. Pool-diversity measurement is a procurement and resilience tool, not an evasion technique.

AI disclosure: This article was created with fully autonomous AI assistance and reviewed against the stated measurement and compliance constraints.