How to Use Crawl4AI for Web Scraping: Complete Setup Guide & Proxy Configuration

Crawl4AI is an open-source web scraping framework designed for developers, supporting asynchronous crawling, JavaScript dynamic rendering, and multiple content extraction methods. When performing batch scraping, frequent requests may trigger 403, 429 errors, or CAPTCHAs. Combining it with proxies can effectively reduce the traffic pressure on a single IP. Using IPFoxy as an example, this guide details how to build a web scraper with Crawl4AI and implement proxy configurations for anti-scraping workflows.

I. What is Crawl4AI Scraper?

Crawl4AI is an open-source web scraping framework based on browser automation. Its core workflow can be summarized as: "Access Webpage → Load Content → Parse Page → Extract Data". Compared with traditional HTTP scrapers, it can handle JavaScript dynamic rendering, making it far more suitable for complex web data collection.

From an architecture perspective, Crawl4AI creates asynchronous crawling tasks via AsyncWebCrawler. The underlying browser visits target URLs, completes page rendering, and extracts HTML, Markdown, or structured data according to specified configurations.

The primary advantages of Crawl4AI include:

• Dynamic Rendering: Supports JavaScript loading, ideal for dynamic web pages.
• Asynchronous Crawling: Handles multiple URLs concurrently to boost batch scraping efficiency.
• Flexible Extraction: Supports CSS and XPath rules, as well as LLM-based structured data extraction.
• Format Conversion: Converts web page content directly into Markdown for easy cleaning, analysis, and AI processing. For simple static pages, traditional HTTP requests are usually sufficient. However, when dealing with dynamic pages, complex structures, or large-scale data collection, Crawl4AI significantly cuts down on browser automation and data parsing overhead.

II. Setting Up Crawl4AI from Scratch: Integrating IPFoxy Proxy Configuration

Below is a complete test tutorial for setting up a web scraper in Python with IPFoxy proxy configuration. Follow these step-by-step instructions:

Step 1: Install Crawl4AI and Prepare Environment

First, create a Python virtual environment and install Crawl4AI:

pip install -U crawl4ai

crawl4ai-setup

The command crawl4ai-setup completes the initialization required for the browser environment. After installation, test your environment with a simple script:

import asyncio from crawl4ai import AsyncWebCrawler

async def main():

 async with AsyncWebCrawler() as crawler:

 result = await crawler.arun("https://example.com")

print(result.markdown[:500])



if name == "main":

asyncio.run(main())

If the terminal outputs the Markdown content of the web page normally, you can proceed with the proxy configuration. If an error occurs at this stage, resolve the Crawl4AI installation or browser dependency issues before adding proxies.

Step 2: Choose Proxy Types Based on Scraping Needs

In Crawl4AI batch web scraping scenarios, a high volume of requests is generated during runtime. Therefore, configuring rotating proxies is necessary to bypass anti-scraping restrictions. Rotating residential proxies are recommended. Compared with static proxies, rotating residential proxies can rotate frequently based on task demands, making them best suited for large-scale public data scraping.

For batch scraping tasks, choosing IPFoxy rotating residential proxies or unlimited residential proxies can effectively reduce data collection failure rates caused by IP issues.

When using IPFoxy rotating residential proxies with Crawl4AI for batch scraping, consider two main rotation modes:

• Sticky Sessions (30-120 minutes): Maintains the same IP connection for a specified timeframe. All requests during this window are routed through the same IP, making it ideal for session-based/cookie-dependent workflows, multi-step tasks, and automated testing.
• Rotating Per Request: Automatically assigns a new residential IP for every single request, suited for high-concurrency and large-scale public data scraping.

In the IPFoxy dashboard, select "Get Dynamic Proxies", then select your target country, proxy type, and rotation mode (sticky or per-request) based on your business requirements.

Step 3: Proxy Configuration

Proxy setup in Crawl4AI is divided into two parts: BrowserConfig manages the browser runtime environment, while CrawlerRunConfig manages the specific crawling task.

BrowserConfig Level: Configure the Browser Create the basic browser configuration (e.g., specifying Chromium and headless mode):

from crawl4ai import AsyncWebCrawler, BrowserConfig

browser_config = BrowserConfig(

    browser_type="chromium",

    headless=True

)

CrawlerRunConfig Level: Configure IPFoxy Proxy Generate and copy your proxy credentials from IPFoxy,

then pass the parameters into ProxyConfig:

from crawl4ai import CrawlerRunConfig, ProxyConfig



run_config = CrawlerRunConfig(

    proxy_config=ProxyConfig(

        server="http://HOST:PORT",

        username="USERNAME",

        password="PASSWORD"

    )

)

Apply Proxy to the Crawling Task Combine both configurations within your scraper task:

async with AsyncWebCrawler(config=browser_config) as crawler:

    result = await crawler.arun(

        url="https://example.com",

        config=run_config

    )

    print(result.markdown[:500])

If result.success returns True and the web page content outputs properly, the proxy setup is working in Crawl4AI.

Step 4: Run and Optimize Scraping Settings

Do not launch large-scale tasks immediately after integration. Run initial tests using 1 to 5 URLs to ensure pages load properly before gradually scaling up your workloads.

If the page opens but requires JavaScript execution to load target content, add wait conditions inside CrawlerRunConfig. For pages with longer load times, adjust wait and timeout parameters accordingly. The rule of thumb is: ensure single-page accuracy first before optimizing concurrency and batch efficiency.

For batch tasks, use multiple proxies and rotate them based on request status. Crawl4AI provides built-in proxy rotation and retry mechanisms to boost overall stability in continuous scraping tasks.

Step 5: Verify Proxy and Scraping Environment

Follow the "IP test first, target web page second" verification sequence: Step 1: Access an IP checker page to confirm that the exit IP matches your chosen IPFoxy proxy region. Step 2: Access the target web page to inspect result.success and verify the returned Markdown or HTML content.

If both checks pass, the proxy integration with Crawl4AI is fully verified. You can now gradually scale up the number of URLs and concurrent tasks. Additionally, store your authentication credentials in environment variables rather than hardcoding them to avoid committing sensitive details to public GitHub repositories.

III. Common Crawl4AI Anti-Scraping Issues and Solutions

Frequent 403, 429 Errors or CAPTCHAs

If 403, 429, or CAPTCHA errors occur frequently during execution, log the request frequency, concurrency level, and proxy IP at the time of failure before making targeted adjustments.

• 403: Indicates that the request was rejected by the target server. Check IP reputation, request headers, and access frequency.
• 429: Indicates rate limiting due to high request frequency. Lower concurrency and increase request intervals.
• CAPTCHA: Indicates enhanced access verification on the target site. Evaluate IP quality and browser environment settings.

To resolve these, reduce concurrency and gradually increase request delays. If restrictions persist, rotate IPs using dynamic residential proxies. Crawl4AI also offers Stealth Mode and Undetected Browser capabilities, though effectiveness depends on the specific detection logic of the target site.

Page Loads

Successfully, but No Content Extracted If the browser accesses the page successfully but target data is missing in the Markdown or HTML, do not switch proxies immediately. This issue usually stems from incomplete dynamic JavaScript loading or mismatched extraction rules.

Troubleshoot in the following order:

• Verify whether target content is dynamically generated via JavaScript.
• Use wait_for to wait for target DOM elements to appear.
• Adjust wait_until based on page loading conditions.
• Check if CSS or XPath selectors remain valid.

Connection Fails Even

After Changing Proxy IPs If connectivity fails across multiple proxy switches, check your configuration before continuing to swap IPs. Follow this troubleshooting order:

• Verify that the Proxy Host and Port are correct.
• Check whether Username and Password credentials are valid.
• Ensure protocol compatibility (e.g., HTTP vs. SOCKS5).
• Test proxy connectivity independently outside the framework.
• Inspect SSL connection handling for HTTPS target pages.

If multiple proxies fail to access the same target URL, the issue likely resides within the Crawl4AI settings, protocol mismatches, or target site policies. Confirming that the proxy connects independently first will significantly speed up debugging in ProxyConfig.

IV. FAQ

Does Crawl4AI require a proxy?

Not necessarily. For small-scale and low-frequency scraping on public pages, a local network connection is sufficient. Proxies become necessary when scaling up operations, targeting specific geographic regions, or handling strict IP rate limits.

What is the benefit of using residential proxies with Crawl4AI?

Residential proxies provide real residential exit IPs for Crawl4AI, helping distribute requests across multiple origins during batch scraping. However, you must still manage request frequencies according to target site policies rather than relying solely on changing IPs.

Why am I still getting 403 errors even after proxy setup succeeds?

A working proxy only confirms that your exit IP has changed; it does not guarantee target site access. You must still manage request rates, IP reputation, browser fingerprints, interaction behaviors, and site-specific access rules.

V. Conclusion

Crawl4AI handles web loading, dynamic rendering, and content extraction, while proxies optimize the underlying network environment for scraping. When executing batch collection, combining dynamic residential proxies with optimal rotation modes—such as per-request rotation or sticky sessions—ensures maximum stability. Always start with small-scale testing before scaling concurrency, IP count, and task parameters.