DEV Community

Jamux Automations
Jamux Automations

Posted on

Building Resilient Web Scrapers in Python: Anti-Detection & Smart Rate Limiting

#python #automation #webdev #productivity

Modern web scraping is no longer about throwing requests.get() at a URL and hoping for the best. With modern Cloudflare turnstiles, Akamai telemetry, and client-side DOM hydration, traditional scraping pipelines fail in seconds.

When we architected our automated freelance radar at Jamux Automations, our polling services needed to operate 24/7 with zero human intervention and 99.9% uptime. Here is the blueprint we use for production-grade scraping in Python.

1. Headless Browser Hardening with Playwright

Standard headless Chromium leaks dozens of telltale flags: navigator.webdriver = true, missing WebGL vendor strings, and abnormal permissions queries.

Here is a hardened initialization snippet using Playwright async in Python:

import asyncio
from playwright.async_api import async_playwright

async def get_stealth_browser():
    p = await async_playwright().start()
    browser = await p.chromium.launch(
        headless=True,
        args=[
            '--disable-blink-features=AutomationControlled',
            '--disable-infobars',
            '--no-sandbox',
            '--disable-dev-shm-usage',
        ]
    )
    context = await browser.new_context(
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36",
        viewport={"width": 1920, "height": 1080},
        locale="en-US",
        timezone_id="America/New_York"
    )

    # Patch navigator.webdriver detection
    await context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
    """)
    return browser, context
Enter fullscreen mode Exit fullscreen mode

2. Exponential Backoff with Jitter

Static time.sleep(3) intervals create predictable behavioral signatures that rate-limiting algorithms detect instantly.

Always implement exponential backoff with randomized jitter:

import random
import time

def resilient_retry(max_retries=4, base_delay=2.0):
    def decorator(func):
        def wrapper(*args, **kwargs):
            delay = base_delay
            for attempt in range(max_retries):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == max_retries - 1:
                        raise e
                    # Full jitter formula
                    sleep_time = random.uniform(0.5, delay)
                    time.sleep(sleep_time)
                    delay *= 2
        return wrapper
    return decorator
Enter fullscreen mode Exit fullscreen mode

3. Real-Time Application: Remote Gig Sniping

We apply these exact principles to our Sniper Leads engine. It monitors remote freelance marketplaces every few minutes, filters contracts with under 25 bids, and crafts AI proposal drafts before human freelancers even open their browsers.

Built by the engineering team at Jamux Automations. What anti-bot challenges are you facing in your scrapers? Let's discuss below.

Top comments (0)