DEV Community: Cenk

Your free AI agent keeps getting IP-blocked: route Crawl4AI and browser-use through a free proxy

Cenk — Thu, 29 Oct 2026 15:23:02 +0000

You built a small agent. It reads pages with Crawl4AI or drives a browser with browser-use, it runs on your laptop or a free-tier VM, and for the first fifty pages it works great. Then the 429s start. Then a site quietly serves you an empty page. Then a cloud IP gets blocked outright.

Most of the time that isn't your code. It's the fact that every request your agent makes leaves from the same IP, and agents make a lot of requests, very fast, with no human pauses. Sites rate-limit per IP, and datacenter IPs (Colab, GitHub Actions, any cheap VPS) start out half-blocked anyway.

The fix is old and boring: spread the requests over many IPs. This post shows how to do that for free, with code for Crawl4AI and browser-use that I ran against the live pool before posting.

Disclosure: I run SolProxy. The free pool below is ours. The code works with any HTTP proxy that takes a username and password; only the address changes.

What a proxy fixes, and what it doesn't

Before the code, be clear about what you're buying (even when it costs $0):

A rotating proxy helps with:

Per-IP rate limits ("too many requests from this IP").
IP bans after your agent hammered one site.
Datacenter IP blocks, when the exit is a home connection.
Geo: seeing the page a visitor from another country sees.

It does not help with:

Serious bot protection (Cloudflare challenges, DataDome, PerimeterX, Amazon). Those look at the browser fingerprint and behaviour, not just the IP. In our own test, the free pool got through on 0 of 36 tries against protected sites. Free proxies won't change that, and paid ones only sometimes do.
Captchas.
Logins. Never send a password or session cookie through a free proxy.

If your agent mostly reads docs, blogs, forums, listings and news, IP rotation is the thing that's missing. If it fights Cloudflare all day, you need more than a proxy.

The free pool in one paragraph

SolProxy's free list re-tests public proxies every minute with a real HTTPS request and drops any that tamper with TLS. With a free account you also get one gateway address, ws.solproxy.net:8501, that picks a live proxy for each connection, only from the ones that passed at least 9 of their last 10 checks, and retries on another one if it dies. You never handle a list.

Limits, so nothing surprises you later: 25 GB per account per month, 2.5 GB per day, 16 connections at a time. The whole free pool also has a monthly budget; if it runs out you get a 503 until the 1st of the month.

Setup: register, copy your token from the dashboard, and use it as the proxy password:

export SOLPROXY_TOKEN=your_token_here
curl -x "http://free:$SOLPROXY_TOKEN@ws.solproxy.net:8501" https://api.ipify.org

You should see an IP that isn't yours.

1. Crawl4AI: pages to LLM-ready markdown, through the pool

Crawl4AI turns a page into clean markdown your model can read. It takes a proxy in BrowserConfig:

pip install crawl4ai
crawl4ai-setup        # installs the browser

# crawl_tool.py
import asyncio
import os

from crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig, ProxyConfig

SOLPROXY = ProxyConfig(
    server="http://ws.solproxy.net:8501",
    username="free",                          # flags go here: "free-us", "free-sticky10m-job1"
    password=os.environ["SOLPROXY_TOKEN"],    # your token from the dashboard
)

BROWSER = BrowserConfig(headless=True, proxy_config=SOLPROXY)
RUN = CrawlerRunConfig(cache_mode=CacheMode.BYPASS, page_timeout=45_000)


async def fetch_markdown(url: str, tries: int = 3) -> str:
    """Fetch a page through the free pool and return LLM-ready markdown."""
    last = "no attempt"
    for _ in range(tries):
        # A new browser means new connections, so each try leaves from another proxy.
        async with AsyncWebCrawler(config=BROWSER) as crawler:
            r = await crawler.arun(url, config=RUN)
        if r.success and (r.status_code or 200) < 400:
            return r.markdown.raw_markdown
        last = f"{r.status_code} {r.error_message or ''}".strip()
    raise RuntimeError(f"{url}: failed after {tries} tries ({last})")


if __name__ == "__main__":
    md = asyncio.run(fetch_markdown("https://news.ycombinator.com"))
    print(md[:500])

fetch_markdown is already an agent tool. Give it to whatever framework you use (LangGraph, the OpenAI or Anthropic tool-calling APIs, smolagents, CrewAI) as "fetch this URL and return its text", and every page your agent reads goes out through a different IP.

Free proxies die mid-request. That's why the retry loop is there: a failed try is cheap, and the next one leaves from a different proxy.

2. browser-use: a full browsing agent behind the pool

browser-use lets an LLM click and type in a real browser. Its Browser takes a proxy too:

pip install browser-use

# agent.py
import asyncio
import os

from browser_use import Agent, Browser, ChatOpenAI
from browser_use.browser.profile import ProxySettings

browser = Browser(
    headless=True,
    proxy=ProxySettings(
        server="http://ws.solproxy.net:8501",
        username="free-sticky10m-agent1",     # keep one exit IP for 10 minutes
        password=os.environ["SOLPROXY_TOKEN"],
    ),
)

agent = Agent(
    task="Open news.ycombinator.com and list the top 5 story titles with their points.",
    llm=ChatOpenAI(model="gpt-4.1-mini"),     # any LLM browser-use supports
    browser=browser,
)

asyncio.run(agent.run())

Note the username. A browsing agent does several steps on one site, and a site that sees the same visitor jump between IPs mid-session gets suspicious. -sticky10m-agent1 keeps the same proxy for up to 10 minutes (or until it dies); the part after it (agent1) is your own session name, so two agents running side by side each get their own pinned proxy. For one-off page reads (the Crawl4AI case), leave it off and let every connection rotate.

3. Plain Playwright

If your agent drives Playwright directly:

import asyncio
import os

from playwright.async_api import async_playwright


async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch(proxy={
            "server": "http://ws.solproxy.net:8501",
            "username": "free",
            "password": os.environ["SOLPROXY_TOKEN"],
        })
        page = await browser.new_page()
        await page.goto("https://api.ipify.org")
        print(await page.inner_text("body"))   # the proxy's IP, not yours
        await browser.close()

asyncio.run(main())

One thing you'll see in a lot of proxy tutorials: ignore_https_errors=True. Don't. HTTPS through an HTTP proxy is an encrypted tunnel; the proxy only sees the hostname. That stays true only while certificate checks are on. If something only works with checks off, a proxy in the chain is swapping certificates and can read your traffic. Our pool drops those, but your browser is the last line of defence, so leave it on.

Useful flags

All flags go in the username, in any order:

Username	What it does
`free`	New proxy per connection
`free-us`, `free-de`	Prefer that country (falls back to any if none is live)
`free-sticky10m-mysession`	Same proxy for up to 10 minutes while it stays up (`sticky1m` up to `sticky1h`)
`free-residential`	Only IPs whose ISP isn't a known hosting company. Returns 503 instead of falling back

How it did when I ran it

I ran the Crawl4AI code and the browser-use proxy setup against the live pool from a home connection before posting (Crawl4AI 0.9.5, browser-use 0.13.14, Playwright 1.64.1). One run, so read it as a sanity check, not a benchmark:

Rotation: four free requests to ipify came out from four different IPs.
Plain sites: Hacker News, Wikipedia, GitHub Trending, quotes.toscrape and books.toscrape all came back as full markdown, in 3 to 6 seconds each. Hacker News failed once with a 503 and worked on the retry, which is exactly why fetch_markdown retries.
Reddit made the browser pass a JavaScript check first, then returned the real page (9 seconds).
Amazon returned a "click to continue shopping" bot page. Crawl4AI still reported it as a success, so don't trust success alone: if the markdown is a few hundred characters, it's probably a block page. Check for text you expect before you hand it to the model.
browser-use: its browser logged in to the pool with the sticky username and loaded pages through it with no extra setup. (The LLM part of the agent doesn't care which proxy is underneath.)

One thing about sticky sessions: free proxies die, and when a pinned one does, the gateway quietly moves you to another live one instead of failing. So "sticky" means "the same proxy for as long as it stays up, up to 10 minutes", not a guarantee.

When free stops being enough

You'll know: you hit the 2.5 GB daily cap, or a target needs a real residential IP that doesn't fall over. The paid pool on the same gateway is $1.00 to $1.20 per GB with no tiers, and the code above doesn't change apart from the username and port. But start free; most agents never need more.

If something in the code breaks with a newer Crawl4AI or browser-use version, leave a comment and I'll fix the post.

Stop wasting time on dead free proxies: a Python setup that actually works

Cenk — Wed, 28 Oct 2026 18:13:57 +0000

If you've ever grabbed a "free proxy list" and pointed your scraper at it, you know what happens next: timeouts, connection resets, and the odd proxy that returns a captcha page instead of the site you asked for.

That's not bad luck. Most free proxies are short-lived. A list that was correct an hour ago is mostly dead now, and a lot of lists only check whether a port is open, which says very little about whether a real HTTPS request will go through.

This post shows a small Python setup that deals with that, in three steps:

Pull a fresh list.
Re-test it yourself with a real HTTPS request.
Rotate through the survivors and retry on failure.

At the end there's a shortcut if you don't want to manage the list at all.

Disclosure: I run SolProxy, the free list used in the examples. The code works with any list that returns ip:port lines.

What makes a free list worth using

Many free lists are copies of other free lists, checked every few minutes to every half hour, and few of them say how they check. When we built the SolProxy list we set a stricter bar:

Re-tested every minute with a real HTTPS request, not a port check. Anything that fails is dropped right away.
TLS tampering is filtered out. Proxies that swap the certificate (and could read your traffic) never make it onto the list.
Uptime you can see. Each proxy shows how many of its last 10 checks it passed, so you can pick stable ones instead of guessing.
More than an IP and port. Every row has the real exit IP, ISP, type, latency and speed.
One endpoint that skips the dead ones for you. With a free account it only picks proxies that passed at least 9 of their last 10 checks. More on that at the end.

Here's how that compares with some popular free lists, based on what each one says on its own page (October 2026):

	SolProxy	free-proxy-list.net	Proxifly	Geonode
How often checked	Every minute	Every 10 to 30 min	Every 5 min	Every minute
Check method published	Real HTTPS request	Not stated	Not stated	Not stated
Filters TLS tampering	Yes	Not stated	Not stated	Not stated
Uptime shown	Yes (last 10 checks)	No	No	Yes
Exit IP and ISP shown	Yes	No	No	ASN only
Latency shown	Yes	No	No	Yes
Free rotating endpoint	Yes (free account)	No (paid)	No (npm module)	No (paid)
SOCKS4/5	No	Paid only	Yes	Yes

If you need SOCKS, the others are the better pick today. If you want HTTPS proxies that were confirmed working a minute ago, or one address that handles the rotation for you, that's where SolProxy is different.

In a small test from a home fiber connection, the endpoint answered 20 out of 20 requests, with a median latency around 1.5 s and no Google captchas. That's a small sample, so take it as a direction, not a benchmark.

1. Pull a fresh list

The list has a plain-text API you can filter by country and type:

import requests

LIST_URL = "https://solproxy.net/free-proxy-list/api.txt"

def get_proxies(country=None):
    params = {"country": country} if country else {}
    text = requests.get(LIST_URL, params=params, timeout=15).text
    return [line.strip() for line in text.splitlines() if line.strip()]

proxies = get_proxies("US")
print(len(proxies), "proxies")

Any other list that gives you ip:port lines works the same way.

2. Re-test before you trust it

Even a list checked a minute ago can contain proxies that died since. Test them yourself, in parallel, with the kind of request you'll actually make:

import concurrent.futures as cf

TEST_URL = "https://api.ipify.org"

def works(proxy, timeout=8):
    url = f"http://{proxy}"
    try:
        r = requests.get(TEST_URL, proxies={"http": url, "https": url}, timeout=timeout)
        return r.ok and r.text.strip() != ""
    except requests.RequestException:
        return False

def alive(proxies, workers=50):
    with cf.ThreadPoolExecutor(workers) as ex:
        results = ex.map(works, proxies)
    return [p for p, ok in zip(proxies, results) if ok]

pool = alive(proxies)
print(len(pool), "passed")

Two things worth knowing:

Test over HTTPS, not HTTP. Plenty of proxies pass plain HTTP and fail on CONNECT.
Keep certificate verification on. requests verifies TLS by default. If a proxy only "works" with verify=False, it is swapping the certificate and can read your traffic. Throw it away.

3. Rotate and retry

Free proxies die mid-job, so every request needs a fallback:

import random

def fetch(url, pool, tries=5):
    for _ in range(tries):
        proxy = random.choice(pool)
        p = f"http://{proxy}"
        try:
            r = requests.get(url, proxies={"http": p, "https": p}, timeout=10)
            if r.status_code < 400:
                return r
        except requests.RequestException:
            pass
        pool.remove(proxy)  # drop dead ones as you go
        if not pool:
            break
    raise RuntimeError("no working proxy left")

r = fetch("https://example.com", pool)
print(r.status_code)

Re-run step 2 every few minutes if your job is long.

The shortcut: one endpoint, no list

If you don't want to manage any of this, SolProxy also has a single free endpoint. It picks a live proxy for each request, only from the ones that passed at least 9 of their last 10 checks, and retries if one dies. The list itself stays open to everyone; the endpoint needs a free account (25 GB a month). Register, copy your token from the dashboard and put it where the password goes:

curl -x http://free:YOUR_TOKEN@ws.solproxy.net:8501 https://api.ipify.org

In Python:

p = "http://free:YOUR_TOKEN@ws.solproxy.net:8501"
r = requests.get("https://api.ipify.org", proxies={"http": p, "https": p}, timeout=15)
print(r.text)  # a different IP than yours

Honest limits

These are public free proxies, and most are datacenter IPs. They're fine for small jobs, testing and geo checks.
Sites with serious bot protection (Amazon, for example) block nearly all of them. Free proxies won't change that.
Never send logins, cookies or anything private through a free proxy.

If something's broken or you'd like a filter that isn't there, leave a comment. I read them all.