Diagnosing Cloudflare Blocks Before Changing Your Scraper

A scraper fails, someone swaps the User-Agent, someone else adds a proxy, then the job starts passing locally but fails again in CI. That usually happens because Cloudflare did not block “scraping” as one thing. It evaluated several signals, and each failure needs a different fix.

This is about authorized automation: your own sites, customer-approved workflows, testing, monitoring, data access you are allowed to perform. If you do not have permission to automate against a site, changing fingerprints or rotating IPs does not make it okay.

Start with the failure you actually see

Cloudflare failures often get collapsed into “403”, but the page body matters.

Common cases:

• Error 1020: usually an access denied page from a Cloudflare rule or bot score decision. The HTTP status may still be 403, so inspect the HTML.
• 403 without a 1020 page: often IP reputation, firewall rules, geo restrictions, or an auth problem.
• 429: rate limit exhaustion. Slowing down can help here, but it will not fix a fingerprint problem.
• Endless Just a moment... page: your client did not complete the browser-side challenge.
• CAPTCHA or Turnstile loop: Cloudflare still considers the session borderline after earlier checks.

Add classification before you add workarounds. Even a basic classifier saves time:

import time
import requests

CLOUDFLARE_MARKERS = {
    "1020": "cloudflare_access_denied",
    "Just a moment": "cloudflare_js_challenge",
    "cf-turnstile": "cloudflare_turnstile",
    "cf-error-code": "cloudflare_error_page",
}

def classify_response(resp: requests.Response) -> str:
    body = resp.text[:5000]

    if resp.status_code == 429:
        return "rate_limited"

    for marker, label in CLOUDFLARE_MARKERS.items():
        if marker in body:
            return label

    if resp.status_code == 403:
        return "forbidden_unknown"

    return "ok" if resp.ok else f"http_{resp.status_code}"


def get_with_backoff(url: str, max_attempts=4):
    for attempt in range(max_attempts):
        resp = requests.get(url, timeout=20)
        reason = classify_response(resp)

        print(resp.status_code, reason)

        if reason == "ok":
            return resp

        if reason == "rate_limited":
            retry_after = resp.headers.get("Retry-After")
            sleep_for = int(retry_after) if retry_after else 2 ** attempt
            time.sleep(sleep_for)
            continue

        # Retrying a 1020 or JS challenge with the same client usually repeats the same failure.
        raise RuntimeError(f"Blocked by {reason}")

    raise RuntimeError("Exceeded retries")

The important bit is the branch that does not retry 1020 forever. If the same HTTP client, same IP, and same timing produced a bot decision, repeating it ten times mostly creates more bad reputation.

Match the fix to the detection layer

Cloudflare can evaluate traffic before your application code ever sees it.

At a high level, you are dealing with three layers:

1. Network and IP reputation: cloud provider ranges, abused proxy pools, and unusual geography can get blocked early.
2. Protocol fingerprinting: TLS and HTTP/2 details can reveal that the client is requests, curl, or a headless browser wrapper rather than a normal browser.
3. Browser execution: JavaScript checks can inspect canvas, WebGL, storage, timing, events, and other browser behavior.

A proxy only changes the network source. It does not make requests execute JavaScript. A headless browser can execute JavaScript, but it will not fix a rate limit if you run 100 sessions in parallel from the same account. A CAPTCHA solver might finish a challenge, but it will not help if your session gets blocked again on the next navigation.

If your scraper has become a queue of browser jobs with blocked-page classification, retry state, and extraction output, Wire fits that job-oriented model better than treating every page as a raw HTTP request.

For 403s tied to IP reputation, residential or ISP routing may help, but it adds cost, latency, and operational risk. You also need sticky sessions when login state matters. Rotating IPs per request can break cookies, CSRF tokens, and fraud systems that expect a user to stay on one network path during a session.

Use browser sessions when the page requires browser behavior

If the response body shows Just a moment..., Turnstile markup, or a 1020 page after initial load, a plain HTTP client is probably the wrong tool. Use Playwright or Puppeteer for the parts of the workflow that genuinely need a browser.

Keep session state separate from credentials. Log in once, save cookies and local storage, then reuse that browser state for later runs:

// login.js
import { chromium } from "playwright";

const browser = await chromium.launch({ headless: false });
const context = await browser.newContext();
const page = await context.newPage();

await page.goto("https://example.com/login");

// Complete login manually or through your approved test credentials.
await page.waitForURL("**/dashboard", { timeout: 120000 });

await context.storageState({ path: "auth-state.json" });
await browser.close();

// scrape-authenticated.js
import { chromium } from "playwright";

const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({ storageState: "auth-state.json" });
const page = await context.newPage();

await page.goto("https://example.com/dashboard", { waitUntil: "domcontentloaded" });
await page.waitForLoadState("networkidle");

const title = await page.locator("h1").textContent();
console.log(title);

await browser.close();

This pattern avoids storing passwords in your automation database. You store session artifacts instead. Treat that file like a secret: encrypt it, scope it per user or tenant, and expire it deliberately.

Browser automation costs more than HTTP scraping. It uses more memory, takes seconds instead of milliseconds, and fails in new ways: missing fonts, different viewport behavior, stale session state, and slow third-party scripts. Use it where the site requires it, not as the default for every URL.

Handle rate limits as a product constraint, not a retry loop

A 429 is the one case where slowing down is the correct first response. Respect Retry-After when it exists. If it does not, use exponential backoff with jitter and cap concurrency per account, per IP, and per target host.

Bad scaling pattern:

100 URLs -> 100 parallel browser sessions -> same account -> same origin -> 429 or account lock

Better pattern:

warm session -> process small batch sequentially -> increase concurrency slowly -> record block rate

Measure these separately:

• success rate by target route
• median and p95 page load time
• number of 1020 pages
• number of 429 responses
• CAPTCHA or Turnstile frequency
• retries per successful extraction

That data tells you whether you have a rate problem, a browser challenge problem, or a session credibility problem.

For your next debugging pass, add response classification and per-layer metrics before changing infrastructure. If 429 dominates, tune concurrency. If 1020 dominates, inspect browser execution. If plain 403s dominate, look at auth, geography, and IP reputation first.