Give Your AI Agent a Web Scraper, Not Just a Browser

Vedaant Singh — Mon, 27 Jul 2026 13:57:46 +0000

AI agents can already browse websites. But browsing is a terrible interface for extracting data.

Browsing gives you a page. Scraping gives you data. A native web fetch drops a rendered page into the model's context and quietly makes the model do the hard part: find the price, parse the table, guess which <div> is the title. That's tokens spent, and it's fragile: reword the page and the extraction breaks.

A scraper does that work for you and returns structured fields ({title, rating, genre, ...}) every time. And there's a second reason that matters even more for the open-source crowd: not every model can browse at all. A local Llama or Qwen running behind an MCP host has no internet unless you give it a tool.

So I added an MCP server to PyScrappy, a Python web-scraping toolkit I maintain, exposing its scrapers as tools that any Model Context Protocol client can call. This post shows how to wire it up, and why the output format is the whole point.

Structured data, not raw pages

The whole argument fits in one picture:

   Browser tool:   agent ─▶ website ─▶ raw HTML ─▶ LLM parses it ─▶ answer
   PyScrappy MCP:  agent ─▶ scraper ─▶ structured JSON ─────────────▶ answer

Everything comes back as structured JSON, so the agent filters, compares, and reasons over it instead of eyeballing HTML. Here's the difference in practice. Ask a browsing agent for a stock quote and it fetches a page like this:

<div class="D(ib) Mend(20px)">
  <fin-streamer value="182.52" ...>182.52</fin-streamer>
  <fin-streamer value="+1.24" ...>+1.24 (0.68%)</fin-streamer>
</div>

...and now the model has to find the number. Ask PyScrappy and you get:

{ "symbol": "AAPL", "price": 182.52, "change": 1.24, "change_pct": 0.68 }

No parsing, no guessing, no wasted context. That's the case for a scraper in one screenshot.

And when you are driving PyScrappy from Python (not through MCP), every result is a ScrapeResult that renders three ways, including clean Markdown you can drop straight into a prompt:

from pyscrappy import scrape

result = scrape("https://en.wikipedia.org/wiki/Web_scraping")

print(result.to_markdown())   # LLM-ready: headings, text, tables
# ...or result.to_json() / result.to_dataframe()

Install

pip install "pyscrappy[mcp]"

That gives you a pyscrappy-mcp command, a stdio MCP server.

Note: the MCP server needs Python 3.10+ (the MCP SDK requires it). PyScrappy's core library still runs on 3.9, but the server does not, so use 3.10 or newer here.

Wire it into your agent

MCP is an open standard, so this works with any MCP-compatible client, not one vendor. That includes Claude (Desktop and Code), editors like Cursor, Windsurf, and VS Code, and open-source hosts such as LibreChat, Cline, and Goose that connect MCP servers to local or open models. The examples below use Claude because it's the fastest way to try it.

Claude Code:

claude mcp add pyscrappy pyscrappy-mcp

Claude Desktop: add to claude_desktop_config.json and restart:

{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp"
    }
  }
}

Cursor / Windsurf / VS Code (Cline) use the same mcpServers block in their own MCP settings file (for example .cursor/mcp.json in Cursor):

{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp"
    }
  }
}

Local / open models (Ollama, etc.): Ollama itself doesn't speak MCP, so you run a host that connects your local model to MCP servers: Goose, Cline, or LibreChat. In Goose, you add the same command as an extension:

extensions:
  pyscrappy:
    type: stdio
    cmd: pyscrappy-mcp

Point that host at an Ollama model that supports tool calling (Llama 3.1, Qwen 2.5, and similar) and it can now pull live web data. This is where the server earns its keep: a model with no web access at all suddenly gets live, structured data.

Ask in plain language

Whichever client you use, you just ask:

"Use pyscrappy to get the latest headlines from bbc.co.uk and the current AAPL quote."

The agent picks scrape_news and scrape_stock, runs them, and reasons over the structured results, with no glue code in between.

What you can do with it

The server ships 22 tools spanning the sources PyScrappy covers:

The open web: scrape any URL into structured text, links, images, and tables (scrape_url).
Reference & research: Wikipedia, dictionary definitions, book search.
Finance & markets: stock quotes, history, and profiles; crypto prices; currency conversion.
News & media: RSS/news feeds, movies and TV (via OMDb), YouTube, SoundCloud.
E-commerce: product search across Amazon, Newegg (tech), and IKEA (home).
Jobs & dev data: public LinkedIn job listings, GitHub repos, Hacker News.
Local & lifestyle: restaurants by city (Zomato, Uber Eats) and weather.

The range shows in a single prompt:

"Use pyscrappy to get today's tech headlines, the current NVDA quote, and a few laptops under $800."

Two tools need a little setup: SoundCloud uses a browser backend (pip install "pyscrappy[browser]"), and the movie tool needs a free OMDb API key passed through the client config:

{
  "mcpServers": {
    "pyscrappy": {
      "command": "pyscrappy-mcp",
      "env": { "OMDB_API_KEY": "your-key" }
    }
  }
}

More tools and sources are on the way, so expect this list to grow.

How it works under the hood

The server is built on the official MCP Python SDK (FastMCP). Each tool is a thin async wrapper around a scraper. PyScrappy's scrapers are synchronous, so each call runs in a worker thread via anyio.to_thread.run_sync to keep the event loop free.

Every tool returns a typed ScrapeToolResult, a stable envelope (data, count, scraper, source_urls, errors) around the source-specific items. That gives clients a declared output schema and validated structured data rather than an opaque string.

Responses are cached briefly (a few minutes), so when an agent asks for the same thing twice, the second call is instant and doesn't hit the site again.

Try it

Repo: https://github.com/mldsveda/PyScrappy
pip install "pyscrappy[mcp]"

AI agents don't need another browser. They need better tools. If you're building agents, give PyScrappy a try and let me know what you'd build with it.

DEV Community: Vedaant Singh