Every geolocation answer on the internet is a guess with a confidence attached.
The label is an inference, not a guarantee
You buy an ISP proxy tagged Chicago, IL. The tag is not a measurement of where traffic leaves the network. It is a row in a geolocation database your vendor licensed and joined to a BGP route. Static ISP exits are genuinely valuable: the ASN resolves to a consumer ISP rather than a hosting provider, so origins treat you as a subscriber, and that is why they survive Cloudflare and Akamai. But the place attached to that trust is a statistical artifact, sold as if it were an address.
So stop reading the label. Observe the exit from more than one vantage point, and dispatch geo-sensitive work only where the evidence converges.
Where the disagreement comes from
Databases disagree because they join the same IP block to different evidence. The MaxMind city tier leans on Wi-Fi positioning and measurement dumps; IP intelligence APIs blend registration data, routing heuristics, and latency triangulation; a CDN only knows which of its edges was closest. Same /24, three answers, sometimes a whole region apart.
Three properties worth internalizing before any code:
- Country is close to ground truth; city is a prior. ASN registration and upstream peering make country placement solid. Subdivision is probable. City is a distribution you sample from.
- Sparse blocks carry enormous centroid error. One representative point for a rural tri-state block puts the city in the middle of nowhere while the exit sits 300 km away.
-
A colo code is not the client's city.
colo=ORDmeans the nearest Cloudflare edge was O'Hare. That is routing topology, not a location assertion; treating it as one is a common bug in scrapers I have reviewed.
The feeds are also self-describing: if a feed says an address is Dallas, every measurement taken from it gets filed as evidence about Dallas. The label reinforces itself and never gets falsified.
Observe from independent witnesses
Probe through the exit and let it tell you where it is. Save this as geo_probe.py. It queries two classes of witness: an IP-intelligence API (a database's opinion, with city and centroid), and the origin side (Cloudflare's cdn-cgi/trace for the colo and observed client country, plus your target's own geo signal, such as Google's regional redirect or a /geo endpoint).
# geo_probe.py -- observed geolocation for one exit
from __future__ import annotations
from dataclasses import dataclass
import requests
TRACE_URL = "https://www.cloudflare.com/cdn-cgi/trace"
# Swap for whichever IP-intel feed you license.
IP_INTEL_URL = "http://ip-api.com/json/?fields=status,country,regionName,city,lat,lon,query"
TIMEOUT = 15
@dataclass(frozen=True)
class Observed:
ok: bool
source: str
country: str = ""
region: str = ""
city: str = ""
lat: float | None = None
lon: float | None = None
colo: str = ""
echo: str = ""
error: str = ""
def _via(proxy: str, url: str, **kw) -> requests.Response:
return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=TIMEOUT, **kw)
def _parse_trace(body: str) -> dict[str, str]:
"""cdn-cgi/trace is key=value lines: ip=, ts=, colo=, loc=, tls=..."""
lines = (line.strip() for line in body.splitlines())
return dict(pair.split("=", 1) for pair in lines if "=" in pair)
def trace_source(proxy: str) -> Observed:
"""Which Cloudflare edge did this exit reach, and what country did it see?"""
try:
f = _parse_trace(_via(proxy, TRACE_URL).text)
return Observed("colo" in f, "cf-trace", country=f.get("loc", ""),
colo=f.get("colo", ""), echo=f.get("ip", ""))
except Exception as exc:
return Observed(False, "cf-trace", error=f"{type(exc).__name__}: {exc}")
def ip_intel_source(proxy: str) -> Observed:
"""Database witness: country, subdivision, city, centroid."""
try:
d = _via(proxy, IP_INTEL_URL).json()
ok = d.get("status") == "success"
return Observed(ok, "ip-intel", country=d.get("country", ""),
region=d.get("regionName", ""), city=d.get("city", ""),
lat=d.get("lat"), lon=d.get("lon"), echo=d.get("query", ""),
error="" if ok else str(d)[:120])
except Exception as exc:
return Observed(False, "ip-intel", error=f"{type(exc).__name__}: {exc}")
def origin_geo_source(proxy: str, geo_url: str = "https://www.google.com/") -> Observed:
"""The target's own signal: Google 302s to a cc host; some sites expose /geo."""
try:
r = _via(proxy, geo_url, allow_redirects=True,
headers={"Accept-Language": "en-US,en;q=0.9"})
cc = ""
if "google." in r.url:
tld = r.url.split("google.", 1)[1].split("/", 1)[0]
cc = {"com": "US"}.get(tld, tld[-2:].upper())
elif r.headers.get("content-type", "").startswith("application/json"):
cc = str(r.json().get("country_code", "")).upper()
return Observed(bool(cc), "origin-geo", country=cc)
except Exception as exc:
return Observed(False, "origin-geo", error=f"{type(exc).__name__}: {exc}")
SOURCES = (trace_source, ip_intel_source, origin_geo_source)
def observed_location(proxy: str, sources=SOURCES) -> dict[str, Observed]:
"""One entry per witness; never a single scalar."""
return {o.source: o for o in (s(proxy) for s in sources)}
The echo field doubles as a leak check: if the IP that comes back is not the exit you configured, the request bypassed your proxy and the reading is worthless.
Score convergence, not the label
The score answers two questions at once: do the witnesses agree with each other, and do they agree with what the task requires? Required-match is a weighted vote over every check a live witness can run: country, state, city, plus a low-weight colo hint. Internal consistency penalises contradictory witnesses. Multiplying them is deliberate: an exit that agrees with itself confidently about the wrong state scores zero, and a scattered exit scores zero regardless of its label.
# geo_router.py -- reconcile witnesses, then dispatch on convergence
from __future__ import annotations
import json
import unicodedata
from dataclasses import asdict, dataclass, field
from time import time
import requests
from geo_probe import Observed, observed_location
US_STATES = {"illinois": "IL", "texas": "TX", "new york": "NY", "california": "CA",
"georgia": "GA", "colorado": "CO", "florida": "FL"}
# Suburbs and boroughs resolve to their metro state.
METRO_OF = {"chicago": "IL", "rockford": "IL", "brooklyn": "NY", "dallas": "TX",
"fort worth": "TX", "oakland": "CA"}
# Nearest edge -> state. Weak: an ORD colo can serve Indiana.
COLO_STATE = {"ORD": "IL", "DFW": "TX", "ATL": "GA", "IAD": "VA", "EWR": "NJ",
"JFK": "NY", "SFO": "CA", "SEA": "WA", "DEN": "CO", "MIA": "FL"}
def _ascii(s: str) -> str:
s = unicodedata.normalize("NFKD", s or "").encode("ascii", "ignore").decode()
return " ".join(s.lower().replace(".", " ").replace("-", " ").split())
def state_of(o: Observed) -> str:
"""Best-effort subdivision; '' means no opinion."""
region, city = _ascii(o.region), _ascii(o.city)
if region in US_STATES:
return US_STATES[region]
if len(region) == 2:
return region.upper()
return METRO_OF.get(city, "")
@dataclass(frozen=True)
class GeoTarget:
city: str
state: str # ISO-3166-2 short code, e.g. "IL"
country: str = "US"
def evidence(target: GeoTarget, o: Observed) -> list[tuple[str, bool, float]]:
"""(check, passed, weight) per runnable check."""
checks: list[tuple[str, bool, float]] = []
if o.country:
checks.append(("country", o.country.upper() == target.country.upper(), 1.0))
state = state_of(o)
if state:
checks.append(("state", state == target.state, 1.0))
elif o.colo.upper() in COLO_STATE:
checks.append(("colo_hint", COLO_STATE[o.colo.upper()] == target.state, 0.4))
if _ascii(o.city):
checks.append(("city", _ascii(o.city) == _ascii(target.city), 0.8))
return checks
def convergence(target: GeoTarget, obs: dict[str, Observed]) -> tuple[float, dict]:
live = [o for o in obs.values() if o.ok]
if not live:
return 0.0, {"internal": 0.0, "flags": ["no witness answered"], "observed": {}}
flags: list[str] = []
countries = {o.country.upper() for o in live if o.country}
states = {state_of(o) for o in live if state_of(o)}
if len(countries) > 1:
flags.append(f"witnesses disagree on country: {sorted(countries)}")
if len(states) > 1:
flags.append(f"witnesses disagree on state: {sorted(states)}")
internal = max(0.0, 1.0 - 0.5 * len(flags))
checks = [c for o in live for c in evidence(target, o)]
total = sum(w for _, _, w in checks)
match = (sum(w for _, ok, w in checks if ok) / total) if total else 0.0
return round(internal * match, 3), {"internal": round(internal, 3),
"required_match": round(match, 3), "flags": flags,
"observed": {o.source: asdict(o) for o in live}}
@dataclass
class Task:
task_id: str
url: str
target: GeoTarget
min_score: float = 0.70
@dataclass
class GeoRouter:
exits: list[str]
ledger: str = "geo_ledger.jsonl"
probes: dict[str, dict[str, Observed]] = field(default_factory=dict)
provenance: dict[tuple[str, str], dict] = field(default_factory=dict)
veto: dict[str, set[str]] = field(default_factory=dict) # state -> exits proven elsewhere
untrusted: set[str] = field(default_factory=set) # self-contradicting exits
def measure(self, proxy: str) -> dict[str, Observed]:
self.probes.setdefault(proxy, observed_location(proxy))
return self.probes[proxy]
def pick(self, task: Task) -> str | None:
rejected = self.veto.setdefault(task.target.state, set())
ready: list[tuple[float, str]] = []
for proxy in self.exits:
if proxy in self.untrusted or proxy in rejected:
continue
score, detail = convergence(task.target, self.measure(proxy))
row = {"task_id": task.task_id, "exit": proxy.split("@")[-1],
"required": f"{task.target.city}, {task.target.state}",
"score": score, "internal": detail["internal"],
"flags": detail["flags"], "observed": detail["observed"],
"probed_at": int(time())}
self.provenance[(task.task_id, proxy)] = row
with open(self.ledger, "a", encoding="utf-8") as fh:
fh.write(json.dumps(row, ensure_ascii=False) + "\n")
if score < task.min_score:
# Wrong place, or incoherent evidence? Only the second is fatal globally.
(self.untrusted if detail["internal"] == 0.0 else rejected).add(proxy)
continue
ready.append((score, proxy))
return max(ready)[1] if ready else None
if __name__ == "__main__":
router = GeoRouter(["http://user:pass@203.0.113.10:8443",
"http://user:pass@198.51.100.11:8443"])
task = Task("seo-il-0001", "https://example.com/search?q=emergency+dentist",
GeoTarget(city="Chicago", state="IL"))
if (proxy := router.pick(task)):
resp = requests.get(task.url, proxies={"http": proxy, "https": proxy}, timeout=20)
row = {"url": task.url, "body_len": len(resp.text),
"_geo": router.provenance[(task.task_id, proxy)]} # evidence travels with data
print(json.dumps(row, ensure_ascii=False))
How a mis-targeted exit quietly corrupts a dataset
The failure is invisible precisely because nothing errors. Your exit is really in Fort Worth while the ledger says Illinois, so the geo=Chicago rows fill with Texas assortment, TX-only promotions, and Illinois results that never surface. A 4% price delta becomes a published finding about regional pricing when it was a routing fact about a /24. Retries make it worse: two attempts through the same exit agree perfectly with each other and disagree with the task, so your QA passes.
That is why min_score is a hard gate rather than a hint. For ad verification and geo-locked pricing, push it to 0.9 and require the state check specifically. For soft regional sampling, 0.6 with country agreement plus a matching colo hint is defensible. If a task genuinely tolerates a 500 km error, say so in the task definition and add a centroid term against the recorded lat/lon instead of discovering the tolerance in a retrospective.
Rules I hold the pool to
- Re-observe on pool refresh and daily. Geo is a property of somebody's routing table, not of your lease. A February route optimisation relocates your "Chicago".
- Cache the verdict, not the score. An exit proven to be in Texas stays vetoed for Illinois tasks without re-probing.
- Quarantine self-contradicting exits globally. If the witnesses disagree, the block is a mixed or NAT'd pool and no city label on it is trustworthy anywhere.
-
Keep
colo, traceloc, and the database city in the provenance blob. They are the cheapest origin-side truth available, and what an auditor asks for. - Reconcile before you blame personalization. Half the "wrong region served" tickets I have seen resolve to an exit in the wrong region.
- Three witnesses for high-stakes geo. Two sources give you a tie, not a consensus.
- Never write a row with empty provenance. A collector that cannot attach its evidence produces rows nobody downstream can defend.
Geolocation is measurement, not an attribute. Keep the error bars next to the data and the class of "the site lied to us" tickets collapses into a boring nightly reconciliation job.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)