Same City, Different Internet: Detecting ISP-Level Content Splits in Your Scraping Pipeline

Most scraping teams geo-target at the country level, some at the city level, and then stop — as if geography fully determined what a server sends you. It doesn't. The network you exit from is part of the equation, and on plenty of targets it's the dominant part.

Last quarter I chased a bug that looked like a parser regression. A pipeline collecting broadband plan data was storing two different prices for the same plan, same URL, same city, captured an hour apart. We assumed an A/B test. Then we looked at the fetch logs and found the real split: the observations came through exits on two different ISPs. The target site — a telecom comparison page, as it turned out — was serving carrier-personalized content: visitors coming from ISP A's address space saw ISP A's plans promoted at the top with an exclusive price, and visitors from ISP B saw a different arrangement entirely. Same city. Same coordinates. Different internet.

The core claim of this article: the request matrix you should be sampling is not (country × city), it's (country × city × ASN). And you don't have to believe carrier-level splits exist on your targets — you can test for them cheaply with a content-diffing harness that fits in one file.

Where ASN-level splits come from

An Autonomous System Number identifies the network that owns your exit IP — the ISP, the mobile carrier, the hosting company. Sites vary content by ASN for reasons that are mostly commercial:

• Carrier landing pages and promotions. Telecoms and streaming services detect the visitor's network and show plan bundles specific to that carrier. This is the textbook case.
• Peering- and CDN-sensitive delivery. Some CDNs and video platforms choose bitrates, formats, or even whole page variants based on network characteristics they infer from the ASN.
• Security tiering. WAFs frequently score requests by the reputation of the network, not just the IP. One ASN can sail through while its neighbor gets challenged — meaning your "geo" test results are silently contaminated by network reputation.
• Regional ISP quirks. Smaller regional networks sometimes get different cached variants or geo-fallback content, because the site's own geolocation of that ASN's ranges is stale or coarse.

The practical consequence for a data pipeline: if you collect from one exit network only, you don't have "the" price, ranking, or catalog for a location — you have that network's version of it. If you're feeding analytics or ML downstream, you've just baked a silent confound into the dataset.

The detection design

The idea is a controlled A/B over networks. Pick one target URL and one location (say, Austin, US). Fetch it repeatedly through exits that share geography but differ in ASN — several major consumer ISPs plus one hosting ASN as a control. Then compare responses after normalization, because raw HTML always differs (timestamps, CSRF tokens, ad slots).

Normalization is where the engineering care goes:

1. Strip or hash-stabilize known-volatile nodes (scripts, nonces, session tokens).
2. Extract the data fields you actually care about (price, plan name, availability) rather than diffing the whole DOM.
3. Hash the normalized field set so equality is cheap and storable.

Then compute, per field, the fraction of ASN pairs on which it differs. Anything above zero is a split candidate; anything that differs only on the hosting-ASN control is more likely a reputation/bot-tier effect than true carrier personalization.

Getting multiple ASNs in one city

This is the part that sounds expensive and isn't, if your proxy provider supports ASN targeting. Instead of requesting "a residential IP in Austin," you request "a residential IP in Austin on network X" by passing the ASN (or carrier name, depending on the provider's parameter scheme) in the proxy credentials. Thordata, which I use for this, lets you specify ASN in the username string the same way you specify country or city, so one account can sample across networks.

Residential pools naturally cover many consumer ISPs per city, which is exactly the population you want — real subscribers on real carrier networks. One caveat: with ASN targeting you have fewer available exits per cell than with plain city targeting, so hold each exit with a sticky session and don't hammer; if a specific (city, ASN) cell has no inventory, the gateway errors immediately, and your harness should record that as "cell unavailable" rather than retry-spamming.

The harness

# asn_split_detector.py -- detect carrier-level content splits for one URL.
# Python 3.8+, stdlib + requests.

import hashlib
import json
import re
import time
from collections import defaultdict
from itertools import combinations

import requests

PROXY_TEMPLATE = (
    "http://{user}-country-us-city-austin-asn-{asn}-session-{sid}"
    ":{password}@gw.thordata.com:8000"
)
PROXY_USER = "youruser"
PROXY_PASS = "yourpass"

# mix of consumer carriers + one hosting network as a control
ASNS = [
    ("AS7922", "comcast"),
    ("AS7018", "att"),
    ("AS22773", "cox"),
    ("AS209",  "centurylink"),
    ("AS15169", "hosting-control"),   # not a consumer ISP
]

TARGET = "https://example-broadband-comparison.example/plans"
REPEATS = 3   # fetches per (asn, session) to separate split from noise

VOLATILE_PATTERNS = [
    re.compile(r"csrf[-_]?token[^\"]*\"[^\"]*\"", re.I),
    re.compile(r"nonce=\"[^\"]+\""),
    re.compile(r"\d{10,13}"),                      # epoch-ish timestamps
]


def proxies_for(asn: str, sid: str) -> dict:
    url = PROXY_TEMPLATE.format(user=PROXY_USER, password=PROXY_PASS,
                                asn=asn, sid=sid)
    return {"http": url, "https": url}


def normalize(html: str) -> str:
    s = html
    for pat in VOLATILE_PATTERNS:
        s = pat.sub("X", s)
    s = re.sub(r"\s+", " ", s)
    return s


def extract_fields(html: str) -> dict:
    """Pull the data fields you care about. Replace with your real parser."""
    fields = {}
    for m in re.finditer(
        r'data-plan="(?P<plan>[^"]+)"[^>]*data-price="(?P<price>[^"]+)"', html
    ):
        fields[m.group("plan")] = m.group("price")
    return fields


def fetch_once(asn: str, sid: str) -> dict:
    resp = requests.get(
        TARGET,
        proxies=proxies_for(asn, sid),
        timeout=30,
        headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"},
    )
    resp.raise_for_status()
    fields = extract_fields(resp.text)
    return {
        "status": resp.status_code,
        "norm_hash": hashlib.sha256(normalize(resp.text).encode()).hexdigest()[:12],
        "fields": fields,
    }


def run():
    samples = defaultdict(list)
    for asn, label in ASNS:
        for rep in range(REPEATS):
            sid = f"{label}-{rep}"          # fresh sticky session per repeat
            try:
                samples[label].append(fetch_once(asn, sid))
            except Exception as e:
                samples[label].append({"error": str(e)})
            time.sleep(3)

    # 1) whole-page split: distinct normalized hashes per ASN
    print("== normalized page hashes ==")
    for label, runs in samples.items():
        hashes = {r.get("norm_hash", "ERR") for r in runs}
        print(f"{label:18s} {sorted(hashes)}")

    # 2) field-level split matrix
    print("\n== field-level differences between ASN pairs ==")
    split_report = {}
    for (l1, runs1), (l2, runs2) in combinations(samples.items(), 2):
        # majority field-set per ASN to guard against single-fetch noise
        def majority(runs):
            counts = defaultdict(int)
            for r in runs:
                key = json.dumps(r.get("fields", {}), sort_keys=True)
                counts[key] += 1
            return max(counts.items(), key=lambda kv: kv[1])[0]
        f1, f2 = majority(runs1), majority(runs2)
        if f1 != f2:
            d1, d2 = json.loads(f1), json.loads(f2)
            diffs = {k: [d1.get(k), d2.get(k)]
                     for k in set(d1) | set(d2) if d1.get(k) != d2.get(k)}
            split_report[f"{l1}|{l2}"] = diffs
    print(json.dumps(split_report, indent=2))

    # 3) control check: hosting ASN vs the consumer-ASN consensus
    consumer = [json.dumps(majority(samples[l]), sort_keys=True)
                for l, _ in ASNS if "hosting" not in l]
    if consumer and len(set(consumer)) == 1:
        ctrl = json.dumps(majority(samples["hosting-control"]), sort_keys=True)
        print("\ncontrol:", "consumer ASNs agree, hosting differs -> "
              "reputation tiering" if ctrl != consumer[0]
              else "consumer ASNs agree, hosting agrees -> no split detected")


if __name__ == "__main__":
    run()

Read the output in three layers. Whole-page hashes differing per ASN with fields identical means layout/personalization chrome only — harmless for data collection. Field-level differences on consumer ASN pairs are the real finding: your dataset is carrier-dependent, and any single-network collection strategy is biased. A hosting-ASN-only difference, with all consumer ASNs agreeing, usually means the split is actually anti-bot tiering, and the fix is exit-quality, not sampling breadth.

What to do with a confirmed split

Once you know a target varies by network, the collection design follows directly. For analytics use cases, sample a fixed panel of ASNs per city and store the ASN on every record, so downstream consumers can group or control for it. For canonical "what is the price" use cases, pin a reference ASN per market and always collect through it — consistency beats representativeness when the number feeds alerts. For ML training data, stratify across ASNs deliberately; a model trained only on one network's variant of a page will fail quietly on all the others.

And re-run detection monthly, quietly, on a few sentinel URLs. Carrier promotions are campaigns — they appear and vanish. The broadband site that split perfectly in June was back to network-agnostic content by August, and the only reason we noticed was the sentinel check.

Geography is a two-dimensional answer to a three-dimensional question. Country and city tell you where the request lands; the ASN tells you which part of the internet it landed on. Sample both, or accept that some fraction of your "location" data was never really about location.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele