Your free AI agent keeps getting IP-blocked: route Crawl4AI and browser-use through a free proxy

You built a small agent. It reads pages with Crawl4AI or drives a browser with browser-use, it runs on your laptop or a free-tier VM, and for the first fifty pages it works great. Then the 429s start. Then a site quietly serves you an empty page. Then a cloud IP gets blocked outright.

Most of the time that isn't your code. It's the fact that every request your agent makes leaves from the same IP, and agents make a lot of requests, very fast, with no human pauses. Sites rate-limit per IP, and datacenter IPs (Colab, GitHub Actions, any cheap VPS) start out half-blocked anyway.

The fix is old and boring: spread the requests over many IPs. This post shows how to do that for free, with code for Crawl4AI and browser-use that I ran against the live pool before posting.

Disclosure: I run SolProxy. The free pool below is ours. The code works with any HTTP proxy that takes a username and password; only the address changes.

What a proxy fixes, and what it doesn't

Before the code, be clear about what you're buying (even when it costs $0):

A rotating proxy helps with:

• Per-IP rate limits ("too many requests from this IP").
• IP bans after your agent hammered one site.
• Datacenter IP blocks, when the exit is a home connection.
• Geo: seeing the page a visitor from another country sees.

It does not help with:

• Serious bot protection (Cloudflare challenges, DataDome, PerimeterX, Amazon). Those look at the browser fingerprint and behaviour, not just the IP. In our own test, the free pool got through on 0 of 36 tries against protected sites. Free proxies won't change that, and paid ones only sometimes do.
• Captchas.
• Logins. Never send a password or session cookie through a free proxy.

If your agent mostly reads docs, blogs, forums, listings and news, IP rotation is the thing that's missing. If it fights Cloudflare all day, you need more than a proxy.

The free pool in one paragraph

SolProxy's free list re-tests public proxies every minute with a real HTTPS request and drops any that tamper with TLS. With a free account you also get one gateway address, ws.solproxy.net:8501, that picks a live proxy for each connection, only from the ones that passed at least 9 of their last 10 checks, and retries on another one if it dies. You never handle a list.

Limits, so nothing surprises you later: 25 GB per account per month, 2.5 GB per day, 16 connections at a time. The whole free pool also has a monthly budget; if it runs out you get a 503 until the 1st of the month.

Setup: register, copy your token from the dashboard, and use it as the proxy password:

export SOLPROXY_TOKEN=your_token_here
curl -x "http://free:$[e-mail]:8501" https://api.ipify.org

You should see an IP that isn't yours.

1. Crawl4AI: pages to LLM-ready markdown, through the pool

Crawl4AI turns a page into clean markdown your model can read. It takes a proxy in BrowserConfig:

pip install crawl4ai
crawl4ai-setup        # installs the browser

# crawl_tool.py
import asyncio
import os

from crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig, ProxyConfig

SOLPROXY = ProxyConfig(
    server="http://ws.solproxy.net:8501",
    username="free",                          # flags go here: "free-us", "free-sticky10m-job1"
    password=os.environ["SOLPROXY_TOKEN"],    # your token from the dashboard
)

BROWSER = BrowserConfig(headless=True, proxy_config=SOLPROXY)
RUN = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, page_timeout=45_000)


async def fetch_markdown(url: str, tries: int = 3) -> str:
    """Fetch a page through the free pool and return LLM-ready markdown."""
    last = "no attempt"
    for _ in range(tries):
        # A new browser means new connections, so each try leaves from another proxy.
        async with AsyncWebCrawler(config=BROWSER) as crawler:
            r = await crawler.arun(url, config=RUN)
        if r.success and (r.status_code or 200) < 400:
            return r.markdown.raw_markdown
        last = f"{r.status_code} {r.error_message or ''}".strip()
    raise RuntimeError(f"{url}: failed after {tries} tries ({last})")


if __name__ == "__main__":
    md = asyncio.run(fetch_markdown("https://news.ycombinator.com"))
    print(md[:500])

fetch_markdown is already an agent tool. Give it to whatever framework you use (LangGraph, the OpenAI or Anthropic tool-calling APIs, smolagents, CrewAI) as "fetch this URL and return its text", and every page your agent reads goes out through a different IP.

Free proxies die mid-request. That's why the retry loop is there: a failed try is cheap, and the next one leaves from a different proxy.

2. browser-use: a full browsing agent behind the pool

browser-use lets an LLM click and type in a real browser. Its Browser takes a proxy too:

pip install browser-use

# agent.py
import asyncio
import os

from browser_use import Agent, Browser, ChatOpenAI
from browser_use.browser.profile import ProxySettings

browser = Browser(
    headless=True,
    proxy=ProxySettings(
        server="http://ws.solproxy.net:8501",
        username="free-sticky10m-agent1",     # keep one exit IP for 10 minutes
        password=os.environ["SOLPROXY_TOKEN"],
    ),
)

agent = Agent(
    task="Open news.ycombinator.com and list the top 5 story titles with their points.",
    llm=ChatOpenAI(model="gpt-4.1-mini"),     # any LLM browser-use supports
    browser=browser,
)

asyncio.run(agent.run())

Note the username. A browsing agent does several steps on one site, and a site that sees the same visitor jump between IPs mid-session gets suspicious. -sticky10m-agent1 keeps the same proxy for up to 10 minutes (or until it dies); the part after it (agent1) is your own session name, so two agents running side by side each get their own pinned proxy. For one-off page reads (the Crawl4AI case), leave it off and let every connection rotate.

3. Plain Playwright

If your agent drives Playwright directly:

import asyncio
import os

from playwright.async_api import async_playwright


async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(proxy={
            "server": "http://ws.solproxy.net:8501",
            "username": "free",
            "password": os.environ["SOLPROXY_TOKEN"],
        })
        page = await browser.new_page()
        await page.goto("https://api.ipify.org")
        print(await page.inner_text("body"))   # the proxy's IP, not yours
        await browser.close()

asyncio.run(main())

One thing you'll see in a lot of proxy tutorials: ignore_https_errors=True. Don't. HTTPS through an HTTP proxy is an encrypted tunnel; the proxy only sees the hostname. That stays true only while certificate checks are on. If something only works with checks off, a proxy in the chain is swapping certificates and can read your traffic. Our pool drops those, but your browser is the last line of defence, so leave it on.

Useful flags

All flags go in the username, in any order:

Username	What it does
free	New proxy per connection
free-us, free-de	Prefer that country (falls back to any if none is live)
free-sticky10m-mysession	Same proxy for up to 10 minutes while it stays up (sticky1m up to sticky1h)
free-residential	Only IPs whose ISP isn't a known hosting company. Returns 503 instead of falling back

How it did when I ran it

I ran the Crawl4AI code and the browser-use proxy setup against the live pool from a home connection before posting (Crawl4AI 0.9.5, browser-use 0.13.14, Playwright 1.64.1). One run, so read it as a sanity check, not a benchmark:

• Rotation: four free requests to ipify came out from four different IPs.
• Plain sites: Hacker News, Wikipedia, GitHub Trending, quotes.toscrape and books.toscrape all came back as full markdown, in 3 to 6 seconds each. Hacker News failed once with a 503 and worked on the retry, which is exactly why fetch_markdown retries.
• Reddit made the browser pass a JavaScript check first, then returned the real page (9 seconds).
• Amazon returned a "click to continue shopping" bot page. Crawl4AI still reported it as a success, so don't trust success alone: if the markdown is a few hundred characters, it's probably a block page. Check for text you expect before you hand it to the model.
• browser-use: its browser logged in to the pool with the sticky username and loaded pages through it with no extra setup. (The LLM part of the agent doesn't care which proxy is underneath.)

One thing about sticky sessions: free proxies die, and when a pinned one does, the gateway quietly moves you to another live one instead of failing. So "sticky" means "the same proxy for as long as it stays up, up to 10 minutes", not a guarantee.

When free stops being enough

You'll know: you hit the 2.5 GB daily cap, or a target needs a real residential IP that doesn't fall over. The paid pool on the same gateway is $1.00 to $1.20 per GB with no tiers, and the code above doesn't change apart from the username and port. But start free; most agents never need more.

If something in the code breaks with a newer Crawl4AI or browser-use version, leave a comment and I'll fix the post.