DEV Community

Greta
Greta

Posted on

When the Exit Changes Country, Your Session Dies: Correlating Exit Geo Drift With Auth Failures

When the Exit Changes Country, Your Session Dies: Correlating Exit Geo Drift With Auth Failures

Your authenticated scraper starts failing at 3 a.m. Not all accounts — a scattered 8%, different times, different origins. The cookies are valid, the token hasn't expired, the code hasn't changed. You check the usual suspects: rate limits, CAPTCHA rates, retry storms. Nothing explains it. Then someone diffs the gateway logs against the failure log and finds the pattern: every failing session was served by an exit whose country had changed since login.

This failure mode is nearly invisible because it lives in the gap between two systems that each report "everything is fine." The proxy gateway reports it delivered an exit — it never promised the same one, or the same geography, forever. The origin reports 403/challenge — it never explains which property of the request it disliked. The session is the only place where the two reports meet, and most scrapers don't record exit metadata at the moment of login, so the comparison is impossible after the fact.

The core idea: authenticated sessions are geo-bounded whether you intended it or not, and the highest-value telemetry you can add is a recorded (login_geo, request_geo) pair on every authenticated request. With that pair, geo drift stops being a mystery and becomes a queryable, alertable metric. This article shows the recording layer, the correlation analysis, and what to do about it.

Why origins care about geo continuity

Origins with meaningful anti-abuse infrastructure don't evaluate requests in isolation; they evaluate session trajectories. A trajectory includes the network layer: ASN, country, and sometimes city-level geography of each IP that presented the credentials. Continuity is expected. A session that authenticated from a Frankfurt residential IP and then presents the same cookies from a Dallas residential IP isn't "a different request" — it's a trajectory break, and trajectory breaks are the canonical signature of credential sharing.

The catch: this check doesn't fire immediately or deterministically. Origins apply it with scoring and hysteresis, which is why the failure is scattered and delayed. Eight percent of accounts failing, hours after the drift, is exactly what a threshold-based trajectory scorer produces.

Residential and mobile pools make this worse in a structural way: exits churn. Mobile exits re-attach to different towers; residential exits get re-assigned by ISP maintenance; a sticky window that expires silently can land your next request on a different exit in a different metro. None of this is a bug in the pool — it's the nature of the product. Your session layer just has to account for it.

Step 1: Record the geo pair at both ends

The recording has to happen twice: once when the session is minted (login), once on every subsequent authenticated request. The login-time record is the one teams skip, and it's the one that makes everything else possible.

import json
import time
from dataclasses import dataclass, asdict

import httpx


@dataclass
class ExitGeo:
    ip: str
    country: str
    city: str
    asn: int
    asn_org: str

    def coarse(self) -> tuple:
        """The granularity origins actually score on."""
        return (self.country, self.asn)


def resolve_exit_geo(client: httpx.Client) -> ExitGeo:
    """Ask a geo endpoint through the CURRENT exit, through the same client
    (and therefore the same proxy binding) used for real traffic."""
    r = client.get("https://ipinfo.io/json", timeout=10.0)
    d = r.json()
    return ExitGeo(
        ip=d["ip"],
        country=d["country"],
        city=d.get("city", ""),
        asn=int(d.get("asn", {}).get("asn", "0").lstrip("AS") or 0),
        asn_org=d.get("asn", {}).get("name", ""),
    )


def record_login_geo(session_id: str, client: httpx.Client, store) -> None:
    geo = resolve_exit_geo(client)
    store.save(
        session_id,
        {"login_geo": asdict(geo), "login_ts": time.time()},
    )
Enter fullscreen mode Exit fullscreen mode

Two implementation details matter more than they look. First, resolve geo through the same client that carries your traffic — a separate HTTP client without the same proxy binding resolves your own egress, which is a silently wrong answer. Second, resolve at login even if it costs one extra request. That request is the cheapest insurance in your entire pipeline; without it, every later comparison is against a guess.

Step 2: Sample the drift, don't assume it

You cannot know an exit drifted without checking, and checking every request through an extra hop is wasteful. The engineering answer is stratified sampling: verify geo on a small random fraction of authenticated requests, plus always on failures.

import random

SAMPLE_RATE = 0.05  # 5% of healthy requests, 100% of failures


def should_verify(ok: bool) -> bool:
    return (not ok) or random.random() < SAMPLE_RATE


def check_geo_drift(session_id: str, client: httpx.Client,
                    request_ok: bool, store) -> dict | None:
    if not should_verify(request_ok):
        return None
    current = resolve_exit_geo(client)
    login = store.load(session_id)["login_geo"]
    login_coarse = (login["country"], login["asn"])
    drifted = current.coarse() != login_coarse
    event = {
        "session_id": session_id,
        "login_coarse": login_coarse,
        "current_coarse": current.coarse(),
        "drifted": drifted,
        "request_ok": request_ok,
        "ts": time.time(),
    }
    store.append_event(event)
    return event
Enter fullscreen mode Exit fullscreen mode

Five percent sampling is enough to estimate a drift rate per session cohort with useful confidence, and it caps the added egress cost at 5% — the kind of overhead you can defend in a review.

Step 3: Correlate, and get the answer in one query

Once events accumulate, the correlation that was invisible in dashboards falls out of a group-by:

from collections import defaultdict


def drift_failure_correlation(events: list[dict]) -> dict:
    rates = defaultdict(lambda: {"fail": 0, "total": 0})
    for e in events:
        key = "drifted" if e["drifted"] else "stable"
        rates[key]["total"] += 1
        if not e["request_ok"]:
            rates[key]["fail"] += 1
    return {
        k: round(v["fail"] / v["total"], 4) if v["total"] else None
        for k, v in rates.items()
    }

# Typical result on a fleet with this problem:
# {"drifted": 0.1132, "stable": 0.0041}
Enter fullscreen mode Exit fullscreen mode

A 27x failure-rate differential between drifted and stable sessions is not an anomaly to explain away — it is the diagnosis. And note what the analysis does NOT require: no access to the origin's internals, no reverse engineering, no guessing at thresholds. It's purely your own telemetry, which is why it survives every origin-side policy change.

Extend the same group-by across two more dimensions and you get the full operational picture: drift-to-failure latency (how long after a geo change does the origin act — that's your countermeasure window), and drift direction (a country change is usually fatal; an intra-country ASN change may be scored more leniently; city changes within the same country and ASN are usually noise).

Step 4: The countermeasures, in order of preference

Correlation gives you the diagnosis; now the treatment options, from best to worst:

  1. Detect and re-pin before the origin notices. When a sampled check detects drift on a healthy session, immediately treat the pairing as broken: drop to anonymous browsing on the new exit, request a fresh exit in the login geography, re-warm, and only then resume authenticated traffic. The origin never sees the trajectory break because you never presented credentials across it. This converts a silent failure into a managed rotation.
  2. Gate the session on a geography predicate. For providers whose sticky endpoint supports geo-qualified session IDs, encode the login country into the session request, so the gateway itself refuses to serve the session from another geography. The drift event then manifests as a gateway error — loud, immediate, and retryable — instead of a delayed origin de-auth.
  3. Fail closed on confirmed drift. If you detect the session already served authenticated requests from a drifted exit, retire the pairing on the assumption the origin has scored it. Sending more authenticated traffic through it just deepens the trajectory damage.

What you should not do is ignore it. The correlation number from Step 3 is also a risk metric for the account itself, not just the scraping job: repeated trajectory breaks can escalate an account's standing with the origin far beyond a single failed request.

Make it a standing alert, not a postmortem

The last step is wiring the correlation into an alert with a threshold. The one-line version: alert when fail_rate(drifted) / fail_rate(stable) > 3 over a trailing 24-hour window. Below that ratio, geo changes are noise; above it, the pool's exit stability has degraded — either the provider changed pool composition, or a specific subnet is churning, or your sticky windows are silently lapsing. All three are actionable, and none are visible from request-level monitoring alone.

The broader lesson generalizes beyond geo: every property of your exit — country, ASN, subnet, even IP reputation band — is part of the session's identity whether you model it or not. The origins model it. The only question is whether your telemetry does. Start with the (login_geo, request_geo) pair; it's twenty lines of code and it converts one of the most confusing failure modes in authenticated scraping into a group-by.


Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele

Top comments (0)