DEV Community

proxybrad
proxybrad

Posted on

I built an open-source MCP server so AI agents can read the web - even sites that block bots

#ai #webdev #programming #proxy

If you've tried to give an LLM agent web access, you've hit this wall: point it at the real web and a plain fetch breaks two ways.

  1. JavaScript sites return an empty <div id="root"> shell — the model gets nothing.
  2. Anything commercially interesting returns a 403 or a Cloudflare "just a moment" page.

I kept hitting this building agent workflows, so I open-sourced the tool I made:
HatFetch — an MIT-licensed MCP server that gives Claude, Cursor, or any MCP
client three tools (scrape, crawl, screenshot) returning pages as clean Markdown.

The idea: escalate only when you have to

Most tools either always spin up a heavy headless browser (slow) or never do (blocked). HatFetch escalates:

  1. HTTP fetch → main-content extraction → Markdown. Fast, no browser.
  2. If the page is an empty JS shell, or gets blocked, it escalates to a real stealth browser (Patchright) running through residential proxies.
  3. If an interactive CAPTCHA still appears and you've set a solver key, it solves it. 
// add to your MCP client (Claude Desktop, Cursor, ...)
{ "mcpServers": { "hatfetch": { "command": "npx", "args": ["-y", "hatfetch"] } } }
Enter fullscreen mode Exit fullscreen mode

Then ask your agent: "read and summarize it."

Being honest about what it can't do

No tool gets past 100% of anti-bot in 2026, and I won't pretend otherwise.
Aggressive Cloudflare / DataDome / PerimeterX still win sometimes. When they do, HatFetch returns an honest error instead of handing your model a CAPTCHA page as if it were content — that silent false-success is, IMO, the real sin in a lot of scraping tooling.

It's reliable on: server-rendered sites, JavaScript apps, geo-restricted pages, and IP-reputation / rate-limit blocks. It's a coin flip on the very hardest targets.

The pieces (all MIT)

  • HatFetch — the MCP server: https://github.com/ProxyHatCom/HatFetch
  • hatbreak — the engine, without MCP: npm i hatbreak
  • scrapy-proxyhat — the same residential proxies as a Scrapy middleware: pip install scrapy-proxyhat

Disclosure: I run ProxyHat, a residential proxy service, and there's a
first-class integration — but everything works with any HTTP proxy via
PROXY_URL, or none at all.

Feedback very welcome, especially on the escalation heuristics — when is it worth spending a browser render?

Top comments (0)