Bisect Your Login Flow Across Two Exits to Find Where Authenticated Scraping Breaks

Here is a bug that eats whole afternoons: your authenticated scraper works perfectly when you run it against one proxy, and returns a login wall when you run it through the pool. Same code, same credentials, same cookies. The response body is polite, the status is 200, and the HTML just … is not logged in. There is no exception to read. You cannot tell if it is the IP, the ASN, the TLS handshake, a CSRF token, or an expiry. So you change one random thing and run it again, which is not debugging, it is slot machines.

The disciplined method is a paired replay: run the identical sequence of requests — including the login itself — through two different exits, a known-good control and the suspect, and diff the observable state (cookies, headers, status, a sentinel "am I logged in?" check) at every step. The first step where the two exits diverge is your root cause, and the shape of the divergence tells you which of four distinct failures you have.

The core idea: stop trying to guess why auth breaks through a proxy pool, and instead localize it by differential replay. Authenticated scraping failures through a pool are almost always one of four classes, and each has a signature you can read off the diff.

Set up a deterministic replay

You need the login flow expressed as an ordered list of steps so you can run the exact same steps through two exits and compare. The do_step function should be pure-ish: same inputs, same request.

from dataclasses import dataclass


@dataclass
class Step:
    name: str
    method: str
    url: str
    needs_csrf: bool = False

    def run(self, session, ctx):
        headers = {}
        if self.needs_csrf and ctx.get("csrf"):
            headers["X-CSRF-Token"] = ctx["csrf"]
        resp = session.request(self.method, self.url, headers=headers)
        # pull anything the next step needs out of this response
        if "csrf" not in ctx:
            ctx["csrf"] = _extract_csrf(resp)
        return resp


def replay(exit_url, steps, creds):
    import requests
    s = requests.Session()
    s.proxies = {"http": exit_url, "https": exit_url}
    ctx = dict(creds)
    trace = []
    for st in steps:
        resp = st.run(s, ctx)
        trace.append({
            "step": st.name,
            "status": resp.status_code,
            "cookies": sorted(s.cookies.keys()),
            "auth_cookie_present": bool(s.cookies.get("session_id")),
            "logged_in": _sentinel_logged_in(resp),   # look for a private marker
        })
    return trace

_sentinel_logged_in matters more than the status code. Ask for a page element that only exists when authenticated — a username in the nav, a private API field, a settings link. A 200 that lacks the marker is a soft-logged-out response, and detecting it by marker rather than by status is the difference between "everything looks fine" and "step 3, right after the POST, is where control says yes and suspect says no."

Run both exits and diff

control = replay(GOOD_EXIT_URL, LOGIN_STEPS, CREDS)
suspect = replay(SUSPECT_EXIT_URL, LOGIN_STEPS, CREDS)

for c, s in zip(control, suspect):
    if (c["status"], c["auth_cookie_present"], c["logged_in"]) != \
       (s["status"], s["auth_cookie_present"], s["logged_in"]):
        print(f"FIRST DIVERGENCE at step '{c['step']}'")
        print(" control:", c)
        print(" suspect:", s)
        break
else:
    print("no divergence — the problem is not this exit")

The first divergence is the only one worth reading. Everything after it is downstream noise. Now map what you saw to a cause.

The four failure classes and their diff signatures

Class A — IP-bound token. Signature: login POST succeeds on both (status 200, auth_cookie_present True), but logged_in goes True -> False on the suspect on the very next request. This is the origin binding the session token to the authenticating IP and then seeing the same token arrive from a different address. In a pool this happens when your sticky window is shorter than your crawl, so the exit rotates mid-flow. Fix is in the proxy layer, not your code: hold the sticky session for the whole authenticated sequence, or re-authenticate after every exit change. Nothing about the login itself is wrong.

Class B — CSRF / anti-replay tied to the handshake. Signature: divergence is at the login POST itself — the GET /login step is identical on both exits, but the POST returns 403 (or a 200 with no cookie) on the suspect while 200-with-cookie on the control. The CSRF token or a nonce was minted under the control's TLS session and the origin expects it to come back over the same one. This is sticky-connection territory: the token is fine, the connection continuity is not. Fix: keep the whole login flow on a single TCP/TLS connection and a single exit; do not let your HTTP client reconnect through a different upstream between the GET and the POST.

Class C — TLS / HTTP fingerprint rejection. Signature: the first GET already diverges — the suspect exit never even gets the login page, or gets a challenge interstitial, while control gets a normal page. The problem is upstream of auth entirely: the origin is rejecting the handshake or the HTTP/2 fingerprint that arrives via that exit's path. The credentials are irrelevant. This is the class people misdiagnose as "my account got banned." Confirm it by replaying with no auth at all; if the control can load the public login page and the suspect cannot, it is the exit, not the session.

Class D — reputation / rate step-up. Signature: identical through login, then a 302 to a captcha/security page appears on the suspect only, on a later request, with no cookie change. The origin authenticated you but does not trust the exit enough for the sensitive action. The diff shows logged_in staying True while your real target request fails — auth succeeded, authorization was downgraded. This is a pool-health problem: that ASN or that specific exit is flagged, and the honest fix is to stop routing sensitive accounts through it, not to code around it.

Why "two exits" and not just "the failing one"

Running the suspect alone tells you it failed. Running control and suspect on the same steps, at the same time, turns a failure into a localization: the control is the answer to "should this step work at all right now?" If control also fails at step 3, you do not have a proxy problem — you have a broken extractor or the origin changed. Half of "my proxies are bad" investigations are actually this, and the paired replay is the cheapest way to rule it out before you touch the pool.

Make the replay reusable and it becomes a health probe: run the control/suspect pair on a schedule and alert only on the first divergent step index. A rising index (divergence creeping later) is a different story than a fixed low index (a class-C handshake wall), and that trend line is far more actionable than raw success-rate percentages.

Caveats that save you from a false positive

Origins are not fully deterministic. A/B tests, CDN edge variance, and load-dependent risk scoring mean a single replay can diverge for reasons that have nothing to do with your proxy. Before you conclude "Class C, handshake wall," run the control/suspect pair three times and require the divergence to appear at the same step on the suspect consistently while the control stays green every time. A one-off 403 on step 2 that never repeats is the origin sneezing; a 403 on step 2 across every suspect run is your bug. Encode that: replay N times, and only trust a divergence that reproduces on the suspect and never on the control.

Two more traps. First, timing: if your replay() fires the suspect immediately after minting the exit, some providers need a moment for the sticky session to bind, so add a short settle delay before the first step or you will "discover" a class-C wall that is really just a not-yet-ready tunnel. Second, connection reuse: run each exit through its own requests.Session (as the code does) and never share the session object, or the control's cookies leak into the suspect and every diff reads "identical" when nothing is.

Fold it into CI as a scheduled probe and alert on the first-divergent-step-index trend rather than raw pass/fail, because a creeping index tells you the origin is tightening its grip before your success rate visibly drops.

Stop guessing which knob. Bisect across two exits and read the divergence.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele