Why most scrapers fail on modern sites (and how we hit 90% success)

anybrowse — Wed, 11 Mar 2026 23:01:35 +0000

Why most scrapers fail on modern sites (and how we hit 90% success)

I've been building web scrapers long enough to notice a pattern: most scrapers quietly fail on 30-40% of the web and return nothing useful.

Not an error. Just a 403, or a Cloudflare challenge page that looks like success but isn't, or an empty HTML shell with no content in it.

Here's what's actually happening.

The three things that kill most scrapers

JavaScript rendering

About 60% of the modern web needs JavaScript to render its content. The classic approach — fire an HTTP request, parse the HTML — returns a blank page. The data you wanted was injected by JS after load.

Libraries like requests and axios don't execute JS. If the site needs it, you get nothing.

Bot detection

Sites like LinkedIn, WSJ, Bloomberg, and most e-commerce platforms run fingerprinting before serving content. They check TLS fingerprint, HTTP header order, canvas and WebGL signatures, and whether JavaScript APIs return expected values.

A headless browser with default settings fails most of these. The TLS fingerprint alone is usually enough to get blocked.

IP reputation

Datacenter IPs are blocklisted on sight by most serious platforms. Cloudflare knows every AWS, GCP, and Azure IP range. Requests from these get challenged immediately, even if you pass the fingerprint checks.

What 90% success actually requires

We built a three-tier fallback for anybrowse:

Direct HTTP — Fast, works on about 50% of sites. Pure HTTP with spoofed headers. No JS, returns in under 2 seconds.
Headless browser with anti-detection — For sites that need JS. Handles viewport randomization, navigator property spoofing, and session warming so the browser doesn't look freshly launched. Covers another 30%.
Real browser over residential proxy — The last resort. Routes through a real Chrome instance on a residential IP. This is what gets through Cloudflare's hardest challenges and paywalls. Slow (10-30s), but it works on things the other two can't touch.

The 90% figure comes from production logs across diverse URLs — news sites, e-commerce, social platforms, paywalls. Not benchmarks on example.com.

What still fails

I'll be straight: 10% of the web still beats us. Login-walled content that requires account age and interaction history, sites that CAPTCHA every request, pages that detect residential proxies by behavioral patterns rather than IP.

If a site requires you to be logged in and browsing for 30 minutes before showing content, no scraper solves that cleanly.

Quick test

curl -X POST https://anybrowse.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://techcrunch.com"}'

No key needed for the first 10 requests per day. The response tells you which tier handled it and how long it took.

If you're building an AI agent that needs to read the web, the MCP config is:

{
  "mcpServers": {
    "anybrowse": {
      "command": "npx",
      "args": ["-y", "anybrowse-mcp"]
    }
  }
}

Bot detection keeps getting more aggressive as AI traffic increases. A single-tier scraper that worked fine two years ago now fails on a big chunk of the web. A fallback chain that tries multiple approaches is the only thing that keeps success rates above 50%.

How I built a scraper that actually works on Cloudflare sites

anybrowse — Sun, 08 Mar 2026 07:15:21 +0000

I was building a research agent. It needed to read news sites, pull earnings reports, scrape job listings. Three hours in, half my URLs were returning empty strings or Cloudflare challenge pages. Not errors. Just nothing useful.

That is when I realized the scraping ecosystem is mostly broken for anything that is not a static blog.

Why scraping keeps failing

There are three things killing most scrapers right now.

JavaScript rendering. A lot of sites ship an empty HTML shell and hydrate via React or Vue. Fetch the URL directly and you get a div with an id and nothing else.

Bot detection. Cloudflare, PerimeterX, DataDome -- they fingerprint your browser. Missing plugins, wrong screen resolution, suspiciously perfect mouse timing. A vanilla Playwright script fails all of these in about 30 seconds.

IP reputation. Datacenter IPs are flagged before your code even runs. AWS, Hetzner, DigitalOcean -- blocked by default on half the sites worth scraping.

You can fight each of these individually. Or you can just not deal with it.

What I built

anybrowse takes a URL and gives you clean Markdown. That is the whole API.

pip install anybrowse

from anybrowse import AnybrowseClient

client = AnybrowseClient()
result = client.scrape("https://techcrunch.com")
print(result.markdown)

Under the hood it runs patched Chromium with randomized fingerprints, falls back to residential ISP proxies when the first attempt fails, and uses a Firefox-based engine (Camoufox) for sites that specifically profile Chrome. CAPTCHA solving is built in via CapSolver.

For AI agents, there is an MCP server that works out of the box with Claude Desktop, Cursor, and Windsurf:

{
  "mcpServers": {
    "anybrowse": {
      "type": "streamable-http",
      "url": "https://anybrowse.dev/mcp"
    }
  }
}

Your agent gets scrape, crawl, search, batch scrape, and structured extraction tools. The search endpoint goes through Brave Search API so it actually returns results instead of timing out on Google.

Honest numbers

90% success rate on general websites. LinkedIn and Twitter are still hard because they require login for most content. Paywalls are a separate problem that scraping does not solve.

The 10% that fails is mostly aggressive per-request CAPTCHAs and strict login walls. CapSolver helps but it is not magic.

Try it

10 free scrapes per day, no signup, no credit card. Credit packs start at $5 for 3,000 scrapes if you need more.

anybrowse.dev | GitHub | Docs

DEV Community: anybrowse

Why most scrapers fail on modern sites (and how we hit 90% success)

Why most scrapers fail on modern sites (and how we hit 90% success)

The three things that kill most scrapers

What 90% success actually requires

What still fails

Quick test

How I built a scraper that actually works on Cloudflare sites

Why scraping keeps failing

What I built

Honest numbers

Try it