Stop Debugging the Whole Crawler: The One-Request Reproduction Loop

Every scraping incident starts the same way. A dashboard goes red at 3am, or someone pings the channel: "the crawler is returning empty pages." The instinct is to pull up the whole pipeline — the scheduler, the queue, the retry policy, the last thousand lines of logs — and scroll. That is the wrong debugging unit. A crawler run is a distributed system on a bad day, and log tails bury the one signal you actually need.

Here is the discipline I want you to adopt instead: you don't understand a "block" until it fits in one reproducible request. The atomic unit of debugging a proxied scraper is not the job, and not the log — it is a single minimal-but-faithful request that you can replay at will, from a known exit, with a known set of headers. Save that request as a fixture. From then on, "are we blocked?" stops being a feeling and becomes a command you run.

The three-step reproduction loop

1. Shrink: capture the failing request, not the failing job

Take the exact URL that came back empty. Take the exact headers that request went out with — user-agent, accept-language, cookie jar, the anti-bot tokens you echo back. And crucially, take the exit it used: which proxy and which session served that specific response.

That last part is why your pipeline has to log which proxy/session handled each response. If your crawler only records "URL X failed," you cannot reproduce anything, because the same URL against two different exits tells two different stories. Tie every response to its egress. When you later replay the request, you want to be able to say honestly "this is the same request that failed, against the same class of exit," or "same request, different exit" — and the difference between those two sentences is the entire diagnosis.

Shrinking means throwing away everything that did not participate in the failure: no scheduling, no queue, no database writes. One function, one request, one saved artifact.

2. Bisect: a 2x2 probe matrix

Now change exactly one variable at a time. The two axes worth locking down for proxied scraping are:

• Exit: direct (your own IP) versus proxied (a residential/mobile exit).
• Client state: fresh (new TLS context, clean user-agent, empty cookie jar) versus persisted (the real browser fingerprint or the session cookie jar the crawler actually used).

That gives you four cells. Replay the shrunk request in each and read the table below. This matrix is illustrative — your specific site will differ — but the shape of the reasoning holds across almost every target.

Probe	Exit	Client state	Typical read
A	direct	persisted	baseline the crawler sees
B	proxied	persisted	baseline via egress
C	direct	fresh	your IP/ASN + identity
D	proxied	fresh	exit class + identity

The four outcomes localize the fault:

• Only-direct-fails (C fails, D passes): your own IP or ASN is blocked. The origin simply does not like where you live on the network. Moving traffic behind a proper residential exit fixes it; touching your code does not.

• Only-proxied-fails (B and D fail, A passes): the origin's rules are punishing that class of exit, or your egress geo is wrong for the content you expect. A datacenter-range exit trips a very different filter than a residential one, and geo-gated pages will silently serve a different shape from the wrong country. This is a config question, not a scraping-logic question.

• Both-fail-fresh-but-one-pair-succeeds (A or B passes, C and D fail): it is a fingerprint or session-state issue, not an IP issue. The persisted cookie jar or the real browser TLS/UA is carrying you through. Your "block" is actually a missing piece of state you forgot to replay.

• Both-fail-always (every cell fails): this is not a block at all. The content, selector, or page schema changed. Stop tuning proxies and go look at the DOM — your sentinel no longer exists because it no longer lives where you think it does.

The value of the matrix is that it collapses an entire "I think we're blocked" thread into one of four concrete answers before you write a line of fix code.

3. Save it: the replayable fixture

A reproduction you can't find tomorrow is a reproduction you did twice. Persist the request and the response essentials as a JSON fixture checked into the repo, next to the code. Redact the proxy credentials — you want the shape of the exit in the fixture, not a secret you'll have to scrub from git history at midnight.

The code

capture_repro fires exactly one request through a proxy and writes a fixture. load_repro reads it back.

import hashlib
import json
import re
from datetime import datetime, timezone

import requests

# Matches user:pass@ inside a proxy URL so we can mask the password.
CRED_RE = re.compile(r"(?P<scheme>https?://)(?P<user>[^:@/]+):(?P<pw>[^@/]+)@")


def redact(proxy_url: str | None) -> str | None:
    if not proxy_url:
        return proxy_url
    return CRED_RE.sub(lambda m: f"{m.group('scheme')}{m.group('user')}:****@", proxy_url)


def capture_repro(url, headers, proxy, out_path, timeout=30):
    """Fire ONE request and save a minimal-but-faithful reproduction fixture."""
    proxies = {"http": proxy, "https": proxy} if proxy else None
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=timeout)
    body = resp.text

    fixture = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "method": "GET",
        "url": str(resp.url),              # post-redirect, the URL that answered
        "headers": dict(headers),
        "proxy": redact(proxy),            # exit class kept, password masked
        "status": resp.status_code,
        "final_url": str(resp.url),
        "redirect_count": len(resp.history),
        "body_length": len(body),
        "body_head": body[:400],           # snippet, for a human glance
        "body_head_sha256": hashlib.sha256(
            body[:400].encode("utf-8", "replace")
        ).hexdigest(),                     # hash, for a machine compare
    }

    with open(out_path, "w", encoding="utf-8") as f:
        json.dump(fixture, f, indent=2, ensure_ascii=False)
    return fixture


def load_repro(path):
    with open(path, encoding="utf-8") as f:
        return json.load(f)

A saved fixture looks roughly like this (illustrative):

{
  "timestamp": "2026-02-18T03:12:44+00:00",
  "method": "GET",
  "url": "https://shop.example.com/p/wireless-earbuds",
  "headers": {"User-Agent": "Mozilla/5.0 ...", "Accept-Language": "en-US,en;q=0.9"},
  "proxy": "http://LWQ-sess7a1f-cc-US:****@gw.example.com:8080",
  "status": 200,
  "body_length": 187342,
  "body_head": "<!doctype html>... price-current ...",
  "body_head_sha256": "9c1f...e0aa"
}

Now the payoff. This pytest loads the fixture, replays it through a fresh rotating exit, and asserts the sentinel is still present. That single test is both block-detection and page-shape regression, and it runs nightly without anyone waking up.

import os
import secrets

import requests

FIXTURE = os.path.join(os.path.dirname(__file__), "fixtures", "product-page.json")
SENTINEL = "price-current"

# Your egress gateway, parameterized by session id.
PROXY_TEMPLATE = os.environ["PROXY_TEMPLATE"]
# e.g. http://USER-sess{sid}-cc-US:[e-mail]:8080


def fresh_exit_url() -> str:
    """A brand-new session id => a brand-new exit for the sticky window."""
    sid = secrets.token_hex(4)
    return PROXY_TEMPLATE.format(sid=sid)


def test_replay_still_shows_sentinel(load_repro):
    repro = load_repro(FIXTURE)
    exit_url = fresh_exit_url()

    resp = requests.get(
        repro["url"],
        headers=repro["headers"],
        proxies={"http": exit_url, "https": exit_url},
        timeout=30,
        allow_redirects=True,
    )

    # Blocked? Redirected to a challenge? Both fail loudly here.
    assert resp.status_code == 200, f"unexpected status {resp.status_code}"
    # Page-shape regression: is our sentinel still on the page?
    assert SENTINEL in resp.text, "sentinel gone: schema change OR blocked exit"
    # Thin-body sanity: a 200 that's 900 bytes is not a real product page.
    assert len(resp.text) > repro["body_length"] * 0.6, "suspiciously thin body"

Notice we deliberately do not reuse the stored (redacted) proxy from the fixture. The fixture records the exit for the historical record; the nightly test rotates to a fresh one on purpose, because a regression test that pins itself to one dying IP becomes a flaky alarm instead of a signal.

Honest caveats

A reproduction is a claim about fidelity, and it lies if you are careless.

Match the geo and the exit class, or the probe lies. Replaying a request that failed on a US residential exit against a EU datacenter exit is not a controlled experiment — you changed two variables at once and the matrix readout is meaningless. Same geo, same class, one thing at a time. This is exactly why the fixture keeps the (redacted) proxy string: it's a reminder of what the original exit was.

Some defenses are time-of-day. A rate limiter that resets at the top of the hour, or a challenge that gets more aggressive during peak crawl traffic, will pass your midday reproduction and fail again at 3am. If your incident has a time signature, capture the fixture at that hour, or capture two and compare.

"Fresh exit" must actually be fresh. If your rotation is really just connection reuse, your "new" probe is the same IP wearing a hat. Use session-id suffix rotation: bump the session token in the proxy username and you get a genuinely new sticky exit each replay, which is precisely what the fresh_exit_url() above is doing. Without real rotation, the whole bisect degenerates.

Keep the fixture next to the code, in version control, with secrets redacted. It becomes a regression asset: every past block and every past schema change leaves behind a tiny, replayable specimen you can pull off the shelf.

Why one request

Your crawler is a distributed system: schedulers, queues, retries, concurrency limits, and a fleet of exits, all changing state under you. Trying to understand a block by watching that system run is like debugging a race condition by staring at the wall clock. A single-request reproduction strips out roughly ninety percent of those variables. What's left is one URL, one header set, one exit — and a verdict. "I think we got blocked" becomes either evidence (the 2x2 matrix names the culprit cell) or a clean config question (geo, exit class, session freshness), and never again a three-hour scroll through logs at 3am. Capture the repro first. The rest is bookkeeping.

Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele