Why Scrapers Still Hit HTTP 429 After Adding Delays

You add sleep(5) between requests, maybe add random jitter, and the scraper still gets HTTP 429 Too Many Requests. The usual mistake is assuming the server only counts timestamps. Many sites count requests against an identity: IP address, session cookie, account token, browser fingerprint, or some combination of those.

A delay only changes one signal

A basic scraper failure often looks like this:

GET /products/123 -> 200
GET /products/124 -> 200
GET /products/125 -> 200
GET /products/126 -> 429 Retry-After: 120

If those requests all share the same identity, spreading them out may not help much. A sliding-window limiter might allow 3 requests per 2 minutes per fingerprint. Your 5-second delay still puts all 4 requests inside the same window.

The same thing happens when you rotate proxies but reuse the rest of the client state. From your side, each request uses a new IP. From the server side, the requests may still share a cookie, TLS signature, user-agent profile, timezone, header order, or localStorage identifier.

For scraping systems where reliable extraction depends on managing IPs, browser fingerprints, and failure handling together, Wire treats those identity details as part of the extraction path rather than something bolted onto a sleep() loop.

What servers usually bucket

Rate limiting usually uses one of two algorithms:

• Sliding window: count requests during the last N seconds.
• Token bucket: refill a fixed number of request tokens over time.

The algorithm matters, but the key detail is the bucket key. Servers rarely keep only one global counter. They can count per:

• IP address
• authenticated account
• API key
• session cookie
• browser fingerprint
• TLS fingerprint
• device or tracking ID stored in localStorage

That means this change may not fix anything:

import time
import requests

for url in urls:
    response = requests.get(url, proxies=next_proxy())
    print(response.status_code, url)
    time.sleep(10)

If requests.get() always sends the same headers, carries the same cookies, or negotiates the same TLS fingerprint, the server can still group the traffic. Worse, if you persist a browser profile between proxy changes, cookies and localStorage can link every new IP back to the same previous session.

Your retry loop should honor Retry-After

When you do get a 429, first read the response. Many servers send a Retry-After header. It can be an integer number of seconds or an HTTP date.

import random
import time
import requests
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_after_seconds(value):
    if not value:
        return None

    try:
        return max(0, int(value))
    except ValueError:
        retry_at = parsedate_to_datetime(value)
        if retry_at.tzinfo is None:
            retry_at = retry_at.replace(tzinfo=timezone.utc)
        return max(0, (retry_at - datetime.now(timezone.utc)).total_seconds())


def get_with_429_backoff(url, session, max_attempts=5):
    for attempt in range(max_attempts):
        response = session.get(url, timeout=30)

        if response.status_code != 429:
            response.raise_for_status()
            return response

        wait = retry_after_seconds(response.headers.get('Retry-After'))
        if wait is None:
            wait = min(60, 2 ** attempt) + random.uniform(0, 1)

        time.sleep(wait)

    raise RuntimeError(f'still rate limited after {max_attempts} attempts: {url}')

This fixes the obvious bad behavior: retrying immediately and making the block worse.

It does not fix identity-based limits. If every retry uses the same cookie jar and browser fingerprint, the server still sees one actor waiting and trying again. Sometimes that is fine. Sometimes the bucket never drains enough for your workload.

Fix the identity lifecycle

If the limit follows the session, manage the session explicitly. Do not let browser state leak across unrelated proxy identities.

With Playwright, that usually means creating isolated contexts and keeping the browser profile internally consistent:

from playwright.async_api import async_playwright


async def fetch_page(url, proxy_url, profile):
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            proxy={'server': proxy_url},
            headless=True,
        )

        context = await browser.new_context(
            user_agent=profile['user_agent'],
            locale=profile['locale'],
            timezone_id=profile['timezone'],
            viewport=profile['viewport'],
        )

        page = await context.new_page()
        response = await page.goto(url, wait_until='domcontentloaded')
        html = await page.content()

        await context.close()
        await browser.close()

        return response.status, html

The important part is not randomizing everything on every request. That can create impossible browser combinations, such as a mobile user agent with a desktop viewport and the wrong timezone for the proxy location. Keep a coherent profile for a session, then discard the context when that session is done.

For repeated extraction against the same URL set, Wire handles job-style URL extraction with retries and cached results, which avoids hitting the origin again for identical work when a cached response is valid.

How to debug the real cause

Do not guess. Log the identity signals you control:

print({
    'status': response.status_code,
    'retry_after': response.headers.get('Retry-After'),
    'proxy_id': proxy_id,
    'user_agent': session.headers.get('User-Agent'),
    'cookie_count': len(session.cookies),
})

Then run small tests:

• Same session, different IP: if 429 continues, the limit is probably not only per-IP.
• New session, same IP: if it clears, cookies or localStorage were likely involved.
• Same headers across many proxies: if all proxies fail together, check fingerprinting.
• Honor Retry-After: if that fixes it, you had a timing problem, not an identity problem.

The practical next step is to add this logging around your current scraper, then change one variable at a time: IP, cookie jar, browser context, and retry timing. If you change all of them at once, you may get past the 429, but you still will not know which part was broken.