DEV Community

Cover image for How to Bypass Anti-Bot Checks with Scrapling in OpenClaw
Preecha
Preecha

Posted on

How to Bypass Anti-Bot Checks with Scrapling in OpenClaw

TL;DR

Scrapling provides two anti-bot fetching modes:

  • StealthyFetcher for Cloudflare-protected and similar sites, with options for Turnstile challenges, canvas fingerprint randomization, WebRTC blocking, browser reuse, and geographic settings.
  • DynamicFetcher for JavaScript-heavy sites, infinite scroll, behavioral checks, and cases where StealthyFetcher is not enough.

You can also integrate Scrapling with OpenClaw to control scraping workflows through natural-language commands.

Introduction

You visit a website to collect data. The page loads. Then you see it: “Access Denied” or a CAPTCHA challenge. The site detected your scraper and blocked access. This scenario affects developers, data scientists, and researchers who need web data for legitimate projects.

Try Apidog today

Websites increasingly use sophisticated anti-bot systems. Cloudflare, PerimeterX, Akamai, and similar services analyze browser fingerprints, behavior patterns, and request characteristics to identify automated access. Traditional HTTP clients often fail against these defenses.

Anti-bot systems can inspect:

  • Browser and device fingerprints
  • Mouse and scroll behavior
  • HTTP headers and TLS characteristics
  • JavaScript execution
  • IP reputation and geographic location

Scrapling provides specialized fetchers for handling these checks. Combined with OpenClaw’s natural-language interface, you can describe the scraping task and let the integration select an appropriate fetching mode.

For API development and testing workflows, Apidog provides tools that complement web scraping by helping you test and validate the data or APIs you work with.

Understanding Anti-Bot Detection

Before choosing a fetching strategy, understand the signals that anti-bot systems evaluate.

Browser fingerprinting

Sites can collect information about a browser, including:

  • Screen resolution
  • Installed fonts
  • WebGL renderer
  • Canvas output
  • Browser and device properties

Automated browsers often expose consistent fingerprints that differ from real user browsers.

Behavioral analysis

Human users move the mouse unpredictably, scroll at different speeds, and type with natural timing. Bots may expose mechanical patterns, such as instant page loads, uniform scrolling, or perfectly timed actions.

Request analysis

Every HTTP request includes headers, TLS fingerprints, and connection characteristics. Standard HTTP libraries such as requests can look different from a real browser.

JavaScript challenges

Modern protection systems execute JavaScript to inspect the browser. Cloudflare Turnstile, for example, runs background checks before allowing access to protected content.

IP reputation

An IP address may be flagged because it belongs to a data center, has a history of suspicious traffic, or does not match the expected geographic location.

Scrapling’s fetchers address these detection vectors through browser automation and configurable evasion features.

Choosing a Scrapling Fetcher

Scrapling provides two primary fetchers for protected sites:

Image

StealthyFetcher

Use StealthyFetcher first for common Cloudflare and similar protections. It uses a browser with built-in evasion techniques and supports options such as:

  • Cloudflare challenge handling
  • Canvas fingerprint randomization
  • WebRTC blocking
  • Google search referer settings
  • Installed Chrome
  • Locale and timezone configuration
  • Proxy configuration

DynamicFetcher

Use DynamicFetcher when the site requires full Playwright automation. It is useful for:

  • Complex JavaScript challenges
  • Infinite scroll
  • Content that loads after page interaction
  • Advanced behavioral checks
  • Manual CAPTCHA intervention

Fetcher selection guide

Scenario Recommended fetcher
Cloudflare protection StealthyFetcher
Turnstile challenge StealthyFetcher
Basic bot detection StealthyFetcher
Complex JavaScript challenges DynamicFetcher
Infinite scroll with anti-bot checks DynamicFetcher
Custom anti-bot solutions DynamicFetcher

Using StealthyFetcher

Basic request

Install Scrapling according to its documentation, then make a request with StealthyFetcher:

from scrapling.fetchers import StealthyFetcher

fetcher = StealthyFetcher()
page = fetcher.get("https://protected-site.com")

print(page.text)
Enter fullscreen mode Exit fullscreen mode

The fetcher attempts to load the page through a browser configured for common anti-detection scenarios.

Handling Cloudflare challenges

Enable Cloudflare challenge handling with solve_cloudflare=True:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://cloudflare-protected-site.com",
    solve_cloudflare=True,
)

print(page.text)
Enter fullscreen mode Exit fullscreen mode

This option is intended for interstitial challenges, such as “Checking your browser before accessing,” and Turnstile widgets.

Randomizing the canvas fingerprint

Canvas fingerprinting can create a unique identifier based on how a browser renders graphics. Enable canvas hiding when a site uses this signal:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    hide_canvas=True,
)
Enter fullscreen mode Exit fullscreen mode

Blocking WebRTC leaks

WebRTC can expose a local IP address even when a proxy is configured. Block WebRTC requests when you need to prevent local IP disclosure:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    block_webrtc=True,
)
Enter fullscreen mode Exit fullscreen mode

Using a Google search referer

Some sites treat traffic from search results differently. You can enable the Google search referer option:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    google_search=True,
)
Enter fullscreen mode Exit fullscreen mode

Using your installed Chrome browser

For a browser fingerprint closer to a normal desktop installation, use the installed Chrome browser:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    real_chrome=True,
)
Enter fullscreen mode Exit fullscreen mode

Matching locale and timezone

Configure the browser’s locale and timezone to match the location you want to use:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    locale="en-US",
    timezone_id="America/New_York",
)
Enter fullscreen mode Exit fullscreen mode

Using DynamicFetcher for Advanced Protection

When a site depends heavily on JavaScript or requires browser interaction, use DynamicFetcher:

from scrapling.fetchers import DynamicFetcher

fetcher = DynamicFetcher()
page = fetcher.get("https://highly-protected-site.com")

print(page.text)
Enter fullscreen mode Exit fullscreen mode

Handling infinite scroll

Infinite-scroll pages require browser automation to trigger additional content loads:

from scrapling.fetchers import DynamicFetcher
from playwright.sync_api import sync_playwright

with sync_playwright() as playwright:
    fetcher = DynamicFetcher(playwright=playwright)
    page = fetcher.get("https://site.com/infinite-scroll")

    page.wait_for_selector(".content-item")

    for _ in range(5):
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        page.wait_for_timeout(1000)
Enter fullscreen mode Exit fullscreen mode

Adjust the number of scroll iterations and delay based on how the target page loads content.

Waiting for JavaScript content

If the initial HTML does not contain the content you need, wait for the network to become idle:

page = DynamicFetcher.get("https://site.com")
page.wait_for_load_state("networkidle")

content = page.content()
Enter fullscreen mode Exit fullscreen mode

This gives the page time to execute its JavaScript and render additional content.

Handling CAPTCHAs manually

Some CAPTCHAs require manual intervention. You can detect the CAPTCHA, save a screenshot, and continue after solving it:

page = DynamicFetcher.get("https://site.com")

if page.is_visible(".captcha-container"):
    page.screenshot(path="captcha.png")

    # Solve the CAPTCHA manually, then continue.
    page.click(".captcha-submit")
Enter fullscreen mode Exit fullscreen mode

For workflows that need an explicit pause, use input from the terminal:

page = DynamicFetcher.get("https://site.com")

if page.is_visible('[class*="captcha"]'):
    page.screenshot(path="manual_captcha.png")
    input("Press Enter after solving the CAPTCHA...")

    page.click(".submit-button")
Enter fullscreen mode Exit fullscreen mode

Controlling Scrapling with OpenClaw

OpenClaw lets you describe Scrapling tasks in natural language. If you need setup instructions, refer to the relevant OpenClaw and Scrapling integration documentation.

Request Cloudflare-protected data 

Get the product data from https://shop.example.com.
This site has Cloudflare protection.
Enter fullscreen mode Exit fullscreen mode

OpenClaw can select StealthyFetcher and enable Cloudflare challenge handling.

Use browser automation for advanced protection 

Scrape the job listings from https://careers.example.com.
Use headless browser mode because the site has strong anti-bot protection.
Enter fullscreen mode Exit fullscreen mode

This directs the workflow toward DynamicFetcher.

Rotate proxies 

Extract data from these 100 URLs and rotate through these proxies:
proxy1.com:8080, proxy2.com:8080, proxy3.com:8080
Enter fullscreen mode Exit fullscreen mode

The workflow can distribute requests across the available proxies.

Spoof a geographic location 

Get the price data from https://site.com using US East Coast settings.
Enter fullscreen mode Exit fullscreen mode

The workflow can configure the browser’s timezone and locale.

Anti-Bot Techniques Explained

Understanding the techniques behind the fetchers helps you choose the right configuration.

TLS fingerprint spoofing

During a TLS handshake, the client sends characteristics that can identify the HTTP client or browser. Standard Python clients often have recognizable fingerprints.

StealthyFetcher and DynamicFetcher use browser-based requests and related evasion techniques to make traffic resemble browser traffic.

User-Agent rotation

Using the same User-Agent for every request can make a scraper easier to identify. Scrapling can rotate User-Agents automatically:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.get("https://site.com")
Enter fullscreen mode Exit fullscreen mode

Header spoofing

Browsers send headers in consistent combinations and orders. Scrapling configures browser requests to match normal browser behavior.

Canvas randomization

A site can inspect the pixels produced by canvas rendering to derive a browser or GPU fingerprint. Canvas randomization adds noise to make the resulting fingerprint less consistent.

Screen resolution and window size

Headless browsers may report default viewport dimensions. Browser configuration can randomize viewport dimensions to better match normal user displays.

Mouse movement simulation

With browser automation, you can add mouse interactions before clicking an element:

page.mouse.move_to_element(".button")
page.mouse.move_by_offset(50, 20)
page.click(".submit")
Enter fullscreen mode Exit fullscreen mode

Use this only when the target workflow requires interaction. Browser automation adds overhead compared with a simple fetch.

Proxy Integration

Proxies can help avoid IP-based blocking and support geographic requests.

Configure a proxy 

from scrapling.fetchers import StealthyFetcher

fetcher = StealthyFetcher()

page = fetcher.get(
    "https://site.com",
    proxy="http://username:password@proxy.example.com:8080",
)
Enter fullscreen mode Exit fullscreen mode

Keep proxy credentials out of source control. Load them from environment variables or a secrets manager in production.

Rotate proxies

For larger scraping jobs, select a proxy for each URL:

import random

from scrapling.fetchers import StealthyFetcher

proxies = [
    "http://proxy1.com:8080",
    "http://proxy2.com:8080",
    "http://proxy3.com:8080",
]

fetcher = StealthyFetcher()

for url in urls:
    proxy = random.choice(proxies)
    page = fetcher.get(url, proxy=proxy)

    # Process page.text or extract the required fields.
Enter fullscreen mode Exit fullscreen mode

Residential proxies

Residential proxies use IP addresses associated with internet service providers rather than data centers. They can be harder to detect, but they generally cost more:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.get(
    "https://site.com",
    proxy="http://residential-proxy-provider:port",
)
Enter fullscreen mode Exit fullscreen mode

Common Anti-Bot Scenarios

Cloudflare

Start with StealthyFetcher and enable Cloudflare handling:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://cloudflare-site.com",
    solve_cloudflare=True,
)
Enter fullscreen mode Exit fullscreen mode

If the site continues showing a challenge, try a longer timeout, installed Chrome, or a proxy with a better IP reputation.

PerimeterX and similar behavioral systems

For behavioral checks that require more browser control, try DynamicFetcher:

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.get("https://perimeterx-site.com")
Enter fullscreen mode Exit fullscreen mode

Akamai

Akamai-protected sites may require a combination of browser settings and proxies:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.get(
    "https://akamai-protected.com",
    proxy="http://residential-proxy:port",
    solve_cloudflare=True,
)
Enter fullscreen mode Exit fullscreen mode

Custom anti-bot systems

For a site with multiple detection signals, combine the available options:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://custom-protected.com",
    solve_cloudflare=True,
    block_webrtc=True,
    hide_canvas=True,
    google_search=True,
    real_chrome=True,
)
Enter fullscreen mode Exit fullscreen mode

If the request still fails, switch to DynamicFetcher and use full browser automation.

Recommended Workflow

1. Start with the simplest fetcher

Begin with a basic request:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.get("https://site.com")
Enter fullscreen mode Exit fullscreen mode

Only enable additional options after you identify a specific failure.

2. Add challenge handling when needed 

if "blocked" in page.text.lower():
    page = StealthyFetcher.fetch(
        "https://site.com",
        solve_cloudflare=True,
    )
Enter fullscreen mode Exit fullscreen mode

For production code, prefer checking the response state or expected content rather than relying only on the word blocked.

3. Respect rate limits

Sending requests too quickly can trigger additional protection:

import time

for url in urls:
    page = StealthyFetcher.get(url)
    time.sleep(2)

    # Process the page.
Enter fullscreen mode Exit fullscreen mode

Use the target site’s documented limits and terms as your baseline.

4. Use suitable proxies in production

Free and low-quality proxies may already have poor reputations. If IP-based blocking is the problem, use an appropriate proxy provider:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.get(
    "https://site.com",
    proxy="http://premium-residential-proxy:port",
)
Enter fullscreen mode Exit fullscreen mode

5. Check robots.txt

Before scraping, inspect the site’s robots.txt:

from urllib.parse import urlparse

domain = urlparse("https://site.com").netloc
robots_url = f"https://{domain}/robots.txt"

print(robots_url)
Enter fullscreen mode Exit fullscreen mode

Also review the site’s terms, access requirements, and applicable laws.

6. Handle failures with a fallback

You can fall back to DynamicFetcher when a basic request fails:

from scrapling.fetchers import DynamicFetcher, StealthyFetcher

try:
    fetcher = StealthyFetcher()
    page = fetcher.get("https://site.com")
except Exception as error:
    print(f"StealthyFetcher failed: {error}")

    fetcher = DynamicFetcher()
    page = fetcher.get("https://site.com")
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

You are still getting blocked

Try the following sequence:

  1. Enable the available evasion options: 
   from scrapling.fetchers import StealthyFetcher

   page = StealthyFetcher.fetch(
       "https://site.com",
       solve_cloudflare=True,
       block_webrtc=True,
       hide_canvas=True,
       google_search=True,
       real_chrome=True,
   )
Enter fullscreen mode Exit fullscreen mode
  1. Switch to DynamicFetcher: 
   from scrapling.fetchers import DynamicFetcher

   page = DynamicFetcher.get("https://site.com")
Enter fullscreen mode Exit fullscreen mode
  1. Add a proxy with a suitable reputation: 
   from scrapling.fetchers import StealthyFetcher

   page = StealthyFetcher.get(
       "https://site.com",
       proxy="http://residential-proxy:port",
   )
Enter fullscreen mode Exit fullscreen mode

Cloudflare challenge loop

If the fetcher remains in a challenge loop, increase the timeout:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    solve_cloudflare=True,
    timeout=120,
)
Enter fullscreen mode Exit fullscreen mode

You can also try the installed Chrome browser:

page = StealthyFetcher.fetch(
    "https://site.com",
    solve_cloudflare=True,
    real_chrome=True,
)
Enter fullscreen mode Exit fullscreen mode

CAPTCHA is not solved automatically

Use DynamicFetcher and pause for manual intervention:

from scrapling.fetchers import DynamicFetcher

page = DynamicFetcher.get("https://site.com")

if page.is_visible('[class*="captcha"]'):
    page.screenshot(path="manual_captcha.png")
    input("Press Enter after solving the CAPTCHA...")

    page.click(".submit-button")
Enter fullscreen mode Exit fullscreen mode

Scraping is too slow

Browser-based anti-bot evasion adds overhead. To improve performance:

  • Prefer StealthyFetcher when it handles the site successfully.
  • Use connection pooling where supported.
  • Choose faster, reliable proxies.
  • Avoid enabling evasion options that are not needed.
  • Reduce unnecessary browser interactions and wait times.

Conclusion

Effective scraping against protected sites starts with identifying the detection signals involved and selecting the appropriate fetcher.

Use this process:

  1. Start with StealthyFetcher.
  2. Enable solve_cloudflare=True for Cloudflare challenges.
  3. Add canvas, WebRTC, browser, locale, or proxy settings only when needed.
  4. Switch to DynamicFetcher for JavaScript-heavy pages and advanced browser interactions.
  5. Add delays, proxy rotation, and error handling for larger jobs.
  6. Check robots.txt, terms of service, and applicable legal requirements.

OpenClaw can provide a natural-language interface for these workflows. Once you collect the required data, Apidog can help you test and validate APIs, build automated test suites, and document endpoints.

FAQ

What is the difference between StealthyFetcher and DynamicFetcher?

StealthyFetcher uses a browser with built-in evasion patches and is generally the faster option for common protections. DynamicFetcher provides full Playwright automation and is better suited to advanced JavaScript challenges and interactive pages, but it can be slower.

Does Scrapling work against every anti-bot system?

No anti-bot solution works in every situation. Scrapling supports common systems such as Cloudflare, PerimeterX, and Akamai, but custom or enterprise systems may require additional configuration or manual intervention.

Is bypassing anti-bot protection legal?

The answer depends on your jurisdiction, the site’s terms of service, and the data or access involved. Public data may be accessible for legitimate purposes, but bypassing authentication or accessing private data without authorization can cross legal boundaries.

Why am I still getting blocked?

Common causes include:

  • Poor IP reputation
  • Requests sent too quickly
  • Insufficient browser or fingerprint configuration
  • JavaScript challenges
  • A custom protection system

Try adding delays, using an appropriate proxy, enabling additional options, or switching to DynamicFetcher.

How should I handle CAPTCHAs?

StealthyFetcher can handle Cloudflare Turnstile in supported scenarios. For other CAPTCHAs, use DynamicFetcher and pause for manual solving or use an authorized CAPTCHA-solving integration.

Can I use my own Chrome browser?

Yes. Set real_chrome=True in StealthyFetcher:

from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    real_chrome=True,
)
Enter fullscreen mode Exit fullscreen mode

Do I need proxies?

Not necessarily for small-scale requests. For production or larger jobs, suitable residential proxies can help reduce IP-based blocking and support geographic requests.

How do I rotate User-Agents?

StealthyFetcher rotates User-Agents automatically. For manual control, pass a custom header:

from scrapling.fetchers import StealthyFetcher

fetcher = StealthyFetcher(
    headers={"User-Agent": "Your-Custom-UA"}
)

page = fetcher.get("https://site.com")
Enter fullscreen mode Exit fullscreen mode

What is the success rate against Cloudflare?

Success depends on the target site, its configuration, your IP reputation, browser settings, and request behavior. No configuration guarantees success for every Cloudflare-protected site. Turnstile and other challenges may still require additional configuration or manual intervention.

Can I scrape from multiple geographic locations?

Yes. Configure the browser’s timezone and locale:


python
from scrapling.fetchers import StealthyFetcher

page = StealthyFetcher.fetch(
    "https://site.com",
    timezone_id="Europe/London",
    locale="en-GB",
)
Enter fullscreen mode Exit fullscreen mode

Top comments (0)