Model Authenticated Scraping Sessions as a State Machine: Catching Illegal Transitions at the Proxy Layer

Most scraping code treats "being logged in" as a boolean. A flag is set after a successful POST /login, cleared on a 401, and that is the whole session model. This works fine for a handful of requests from a single exit. It falls apart the moment you run authenticated collection through a rotating residential pool across many accounts, because the boolean cannot express the failure modes that actually happen: a session that silently drops back to anonymous without a logout, an auth cookie that outlives the sticky exit that minted it, a request that escapes the affinity window and lands on a different IP mid-flow.

The fix is not more flags. It is to model the session as a finite state machine (FSM) with explicit states and guarded transitions, and to make the proxy layer part of the state. This article shows a small, runnable Python implementation and, more importantly, the one core idea behind it: an illegal state transition is a far more reliable signal of a proxy-affinity leak than any status code you can inspect after the fact.

The states, and why the proxy belongs in them

A minimal but honest session lifecycle for an authenticated scraper has five states:

• ANONYMOUS — no auth material, or auth material that the origin no longer recognizes.
• PENDING — a login request has been issued but not yet confirmed.
• AUTHENTICATED — the origin returned valid credentials and we hold a live token/cookie set.
• EXPIRING — the token is inside its refresh window; requests still work but must be re-validated.
• DEAD — terminal. Requires a fresh login; the account/exit pairing that produced it is retired.

The crucial extra field is not a state at all — it is an invariant: every request issued while AUTHENTICATED must leave from the same exit that carried the login. The moment that invariant is violated, the origin sees credentials presented from an IP it never authenticated, which is exactly the pattern that triggers step-up challenges and silent de-authentication.

So a session is (state, exit_id, issued_at, expires_at). Transitions are functions that move between states and assert the exit invariant.

A tiny FSM you can actually run

from dataclasses import dataclass, field
from enum import Enum, auto
from time import monotonic


class State(Enum):
    ANONYMOUS = auto()
    PENDING = auto()
    AUTHENTICATED = auto()
    EXPIRING = auto()
    DEAD = auto()


class IllegalTransition(Exception):
    """Raised when a transition is not allowed from the current state,
    or when the exit-affinity invariant would be violated."""


# legal: current state -> set of states we may move to
LEGAL = {
    State.ANONYMOUS:   {State.PENDING},
    State.PENDING:     {State.AUTHENTICATED, State.ANONYMOUS, State.DEAD},
    State.AUTHENTICATED: {State.EXPIRING, State.DEAD},
    State.EXPIRING:    {State.AUTHENTICATED, State.DEAD},   # refresh or die
    State.DEAD:        set(),
}


@dataclass
class Session:
    account: str
    state: State = State.ANONYMOUS
    exit_id: str | None = None
    issued_at: float = 0.0
    expires_at: float = 0.0
    events: list[str] = field(default_factory=list)

    def _move(self, to: State, exit_id: str):
        if to not in LEGAL[self.state]:
            raise IllegalTransition(
                f"{self.account}: {self.state.name} -> {to.name} is not legal"
            )
        # exit-affinity invariant: authenticated traffic must stay on one exit
        if self.state in (State.AUTHENTICATED, State.EXPIRING) and self.exit_id != exit_id:
            raise IllegalTransition(
                f"{self.account}: auth traffic moved {self.exit_id} -> {exit_id}"
            )
        self.events.append(f"{self.state.name} -> {to.name} @ {exit_id}")
        self.state = to

    def begin_login(self, exit_id: str):
        self._move(State.PENDING, exit_id)
        self.exit_id = exit_id

    def confirm_login(self, exit_id: str, ttl: float):
        self._move(State.AUTHENTICATED, exit_id)
        self.exit_id = exit_id
        self.issued_at = monotonic()
        self.expires_at = self.issued_at + ttl

    def note_request(self, exit_id: str):
        # called before every authenticated request
        if monotonic() >= self.expires_at:
            self._move(State.DEAD, self.exit_id or exit_id)
            return False
        if monotonic() >= self.expires_at - 60:
            if self.state is State.AUTHENTICATED:
                self._move(State.EXPIRING, self.exit_id)
        self._move(self.state, exit_id)  # reuses the affinity assertion
        return True

note_request is the workhorse. It is called on the hot path, before every request, and it does two cheap things: advance AUTHENTICATED -> EXPIRING as we approach the token TTL, and enforce that the current request's exit still matches the login exit. Nothing about it touches the network, so you can run it millions of times a second inside a crawler and still pay almost nothing.

Why this catches what status codes miss

Consider the classic incident: a worker reuses a pooled requests.Session across two accounts. The origin returns 200 OK for a stretch, because it is serving a cached or lightly-guarded page, and only later flips to a login wall. By then your logs are full of 200s and the correlation to the actual IP swap is gone.

With the FSM, the swap is caught at the first request after it happens, independent of the response. note_request raises IllegalTransition because self.exit_id != exit_id while the state is still AUTHENTICATED. The status code is irrelevant; the invariant is the alarm.

The second failure the FSM makes visible is the boolean can never see at all: a session that goes AUTHENTICATED -> ANONYMOUS. In LEGAL that transition simply does not exist. You cannot "quietly" fall back to anonymous while authenticated — you must pass through DEAD first, which forces you to retire the account/exit pairing and re-login deliberately. That single constraint turns a whole family of "why did half my rows come back as the logged-out page?" bugs into loud exceptions at their source.

Wiring the exit into the state

The exit_id is the bridge between the FSM and your proxy layer. In a rotating-pool design, an "exit" is not just a hostname:port; it is a sticky session that the provider holds for a bounded window. Two practical notes:

1. Mint the exit_id from the sticky session identifier your provider gives you (often embedded in the proxy URL path or username), not from the resolved public IP. Providers rotate the underlying IP under a sticky id; if you key affinity on the IP you will see phantom transitions and false alarms.
2. When confirm_login records expires_at, set it to min(token_ttl, sticky_window_remaining). There is no point claiming a session is AUTHENTICATED for ten minutes if the exit that makes it valid evaporates in ninety seconds. Derive the number from the provider's sticky window, not from your optimism.

A small scheduler keeps everything honest:

def dispatch(sessions, exit_resolver, do_request):
    ok, leaked = [], []
    for s in sessions:
        exit_id = exit_resolver(s.account)          # sticky id for this account
        try:
            alive = s.note_request(exit_id)
            if not alive:
                s.begin_login(exit_id)              # re-login on a fresh exit
                continue
            resp = do_request(s, exit_id)
            ok.append((s.account, resp.status_code))
        except IllegalTransition as e:
            leaked.append(str(e))                    # the money signal
            s.state = State.DEAD                     # quarantine the pairing
    return ok, leaked

leaked is the list you page on. Every entry is a real affinity breach: authenticated traffic that was about to leave the exit that justified it. In production this metric ("illegal transitions per hour") is the single health number most worth keeping for a multi-account, proxy-fronted pipeline, because it rises before your data quality visibly degrades.

What this buys you over the boolean

The state machine does not make your logins succeed more often — the origin decides that. What it does is make two things deterministic: you always know which account/exit pairing is still trustworthy, and you get a hard error, not a silent wrong answer, the instant authenticated traffic drifts off its exit. For a beginner that mostly means fewer ghosting bugs. At fleet scale, with rotation and sticky windows colliding across dozens of accounts, it is the difference between "some rows are subtly logged-out and we found out three days later" and "we caught the leak on request #1,412 and quarantined that exit."

Model the session. Put the proxy in the state. Let the illegal transitions be your alarm system.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele