One URL, Four Countries, Four Answers: A Controlled Geo-Divergence Experiment in Python

"Does this site serve different content by country?" is the kind of question engineering teams answer with folklore. Someone remembers seeing a currency switch once; someone else is sure the site is fully localized; a third person insists the redirect only happens for EU traffic because of consent rules. The dataset has none of this, because nobody ever ran the experiment.

The core claim of this article: geo-divergence is not a yes/no property, it's a per-field, per-country-pair measurement, and you can get a defensible one for any site with an afternoon of Python, four country-targeted proxy exits, and a diff harness. What's more, measuring it changes what you build: the field-level breakdown tells you which parts of a page are safe to treat as location-independent and which parts make your dataset lie if you ignore geography.

The experiment design

Keep it genuinely controlled, or the results are noise:

• Fixed URL set. Pick 10–30 representative URLs (product pages, landing pages, category pages) and freeze the list for the whole run.
• Fixed vantage points. Four countries with clean separation and interesting divergence potential — I'll use US, UK, DE, JP. Fetch every URL from every country, with repeats so you can separate geography from time.
• One fetch window. Run all four countries' passes close together in time, interleaved, so a content deployment mid-experiment doesn't masquerade as a geo difference. Interleaving (US-1, UK-1, DE-1, JP-1, US-2, UK-2, ...) is the cheap insurance.
• Stable exits. Use a sticky session per country per run so each pass is one coherent vantage point, not a rotating sample of different cities. Country-level intent, one session per (country, repeat) cell.
• Normalization before comparison. Volatile junk — CSRF tokens, timestamps, cache-busters, ad slots — gets stripped before hashing. You're measuring meaningful divergence, not byte noise.

The output I want per URL is a matrix like:

              US→UK   US→DE   US→JP   UK→DE   UK→JP   DE→JP
status           =      =      =       =      =       =
final_url        =      ≠      ≠       ≠      ≠       =
title            =      ≠      ≠       ≠      ≠       ≠
price_fields     ≠      ≠      ≠       ≠      ≠       ≠
stock_fields     =      =      ≠       =      ≠       =
html_norm        ≠      ≠      ≠       ≠      ≠       ≠

Read it top to bottom: redirects and titles diverge (locale routing), prices diverge everywhere (currency + market pricing), stock diverges only for JP (regional inventory), and normalized HTML differs across the board (localized chrome). That's a fingerprint of the site's localization architecture — and each row implies a different collection decision downstream.

Getting the vantage points

For the exits: country-targeted residential proxies. With Thordata's gateway, the country goes inline in the proxy username (...-country-de-...), plus a session token to pin the exit, so each vantage point is one line of configuration. The important behavior to verify early: your exit for each country actually geolocates there. Log the exit IP per fetch and check it once — a mislabeled vantage point poisons every comparison in that column.

The harness

# geo_divergence.py -- controlled 4-country content divergence experiment.
# Python 3.8+, stdlib + requests.

import hashlib
import json
import re
import time
from collections import defaultdict
from itertools import combinations

import requests

COUNTRIES = ["us", "uk", "de", "jp"]
REPEATS = 3
URLS = [
    "https://example-shop.example/product/alpha",
    "https://example-shop.example/product/beta",
    "https://example-shop.example/category/tools",
    # ... freeze this list for the whole experiment
]

PROXY_TMPL = ("http://youruser-country-{cc}-session-{sid}:yourpass"
              "@gw.thordata.com:8000")

VOLATILE = [
    re.compile(r"csrf[^\"']*[\"'][^\"']*[\"']", re.I),
    re.compile(r"nonce=\"[^\"]+\""),
    re.compile(r"\b\d{13}\b"),                     # ms timestamps
    re.compile(r"cache[a-z]*=\d+", re.I),
]


def proxies(cc, sid):
    url = PROXY_TMPL.format(cc=cc, sid=sid)
    return {"http": url, "https": url}


def normalize(html):
    s = html
    for pat in VOLATILE:
        s = pat.sub("X", s)
    return re.sub(r"\s+", " ", s)


def extract(html):
    """Field extraction: the parts that are *data*, not chrome."""
    return {
        "title": (re.search(r"<title>(.*?)</title>", html, re.S) or [None, ""])[1].strip(),
        "currency": sorted(set(re.findall(r'data-currency="([A-Z]{3})"', html))),
        "price": sorted(set(re.findall(r'data-price="([\d.,]+)"', html))),
        "stock": sorted(set(re.findall(r'data-stock="(\w+)"', html))),
    }


def fetch(url, cc, sid):
    r = requests.get(url, proxies=proxies(cc, sid), timeout=30,
                     headers={"User-Agent": "Mozilla/5.0"},
                     allow_redirects=True)
    return {
        "status": r.status_code,
        "final_url": r.url,
        "fields": extract(r.text),
        "norm_hash": hashlib.sha256(normalize(r.text).encode()).hexdigest()[:12],
    }


def run():
    # interleave countries per repeat so time effects can't mimic geo effects
    observations = defaultdict(lambda: defaultdict(list))   # url -> cc -> [obs]
    for rep in range(REPEATS):
        for url in URLS:
            for cc in COUNTRIES:
                sid = f"{cc}-r{rep}"
                try:
                    observations[url][cc].append(fetch(url, cc, sid))
                except Exception as e:
                    observations[url][cc].append({"error": str(e)})
                time.sleep(1.5)

    report = {}
    for url, by_cc in observations.items():
        consensus = {}       # majority value per country, per aspect
        for cc, obs_list in by_cc.items():
            good = [o for o in obs_list if "error" not in o]
            if not good:
                continue
            def maj(key, pick):
                seen = defaultdict(int)
                for o in good:
                    seen[pick(o, key)] += 1
                return max(seen.items(), key=lambda kv: kv[1])[0]
            consensus[cc] = {
                "status": maj("status", lambda o, k: o.get(k)),
                "final_url": maj("final_url", lambda o, k: o.get(k)),
                "norm_hash": maj("norm_hash", lambda o, k: o.get(k)),
                **{f"field:{k}": maj(k, lambda o, k: json.dumps(
                    o.get("fields", {}).get(k), sort_keys=True))
                   for k in ("title", "currency", "price", "stock")},
            }

        pair_matrix = {}
        for a, b in combinations(COUNTRIES, 2):
            if a not in consensus or b not in consensus:
                continue
            row = consensus[a]
            col = consensus[b]
            pair_matrix[f"{a}->{b}"] = {
                k: ("=" if row[k] == col[k] else "≠")
                for k in row
            }
        report[url] = pair_matrix

    # aggregate: how often does each aspect diverge across any pair?
    aspect_divergence = defaultdict(int)
    total = 0
    for url, matrix in report.items():
        for pair, aspects in matrix.items():
            total += 1
            for k, v in aspects.items():
                if v == "≠":
                    aspect_divergence[k] += 1
    print(json.dumps(
        {"per_url": report,
         "aspect_divergence_rate": {k: round(v / total, 3)
                                    for k, v in aspect_divergence.items()}},
        indent=2, default=str))


if __name__ == "__main__":
    run()

Two pieces of the harness do disproportionate work. The majority-vote consensus per country (over repeats) is what separates durable geography-driven differences from one-off noise and A/B flicker — if a field differs within a country across repeats, treat it as volatile and exclude it from the geo conclusion. And the interleaved run order means a mid-experiment deploy hits all four countries roughly equally, so it shows up as a within-country change rather than a between-country one.

Reading the results like an engineer

The interesting outcomes cluster into recognizable patterns:

• Currency-only divergence. Prices differ by country but stock and availability match. The site has market pricing but unified inventory. For collection: one canonical field set per country is enough; you can't synthesize a "true" price, so store price + country as the unit.
• Redirect topology. final_url differing tells you the site routes by locale path (/de/..., /ja/...) or by TLD. Now your URL list is country-specific, and "the same product" needs an ID-level join, not a URL join — a common silent bug in multi-country datasets.
• Fields that never diverge. The most valuable row in the matrix. Content that is provably location-independent (say, spec tables) can be fetched from a single cheap vantage point, and only the divergent fields need per-country collection. That's often a 3–4× bandwidth reduction on a large crawl.
• Status-code divergence. A 200 in one country and a 403/redirect-to-consent in another isn't content localization, it's access architecture — geo-fencing or compliance walls. It tells you which vantage points your pipeline genuinely needs versus which are cosmetic.

One honest limitation to keep in mind: four countries is a measurement, not a census. It establishes that divergence exists and characterizes its shape on the fields you extract; it does not enumerate every market variant. Extend the matrix with a fifth market when a downstream consumer actually needs that market, not preemptively — the harness makes adding one a ten-line change.

The habit worth keeping

Run this experiment once per target before you build its collector, and re-run it quarterly against a frozen sentinel URL list. Localization architectures change — sites add markets, tighten geo-fences, restructure locale routing — and a collector designed against last quarter's divergence matrix fails quietly, writing mismatched fields into your dataset instead of erroring. A 30-URL × 4-country × 3-repeat run is a few hundred requests; treating it as a standing integration test for "does my mental model of this site still hold" is one of the cheapest quality investments in the whole pipeline.

Geography is not a box you check at fetch time. It's a variable the target site controls, and this experiment is how you take the measurement back.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele