Add web scraping to Claude Desktop in 60 seconds (MCP + CrawlAPI)

#ai #mcp #claude #webdev

Claude Desktop is powerful. But by default it can't browse the web.

Here's how to fix that in 60 seconds using the CrawlAPI MCP server — no coding required.

What you get

Three tools available directly in Claude (and any MCP-compatible editor like Cursor or Windsurf):

scrape_url — fetch any page as clean markdown
batch_scrape — scrape up to 10 URLs in parallel
search_and_scrape — search the web + scrape top results in one shot

Setup

1. Get a free API key

Head to RapidAPI and subscribe (free tier: 50 calls/day, no credit card).

2. Download the MCP server

git clone https://github.com/JDWolf96/crawlapi-js
cd crawlapi-js
npm install

3. Add to Claude Desktop config

Open ~/Library/Application Support/Claude/claude_desktop_config.json (Mac) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "crawlapi": {
      "command": "node",
      "args": ["/absolute/path/to/crawlapi-js/mcp-server.js"],
      "env": {
        "CRAWLAPI_KEY": "your_rapidapi_key_here"
      }
    }
  }
}

4. Restart Claude Desktop

That's it. You'll see the CrawlAPI tools appear in the tools panel.

What it looks like in practice

Ask Claude:

"Scrape https://news.ycombinator.com and summarise the top 5 stories"

"Search for 'LangChain vs LlamaIndex 2026' and give me a comparison"

"Batch scrape these 5 competitor pages and tell me how they position themselves"

Claude uses the tools automatically — you just ask in plain English.

Cursor / Windsurf / Continue

Same config, different location. Check your editor's MCP settings panel. The server works with any stdio-transport MCP client.

Why markdown matters

Most scraping APIs return raw HTML. CrawlAPI returns clean markdown — no nav bars, no cookie banners, no scripts. Just the content, ready to drop into an LLM context window.

It also handles JavaScript rendering, so pages that load content dynamically (SPAs, React apps, etc.) work out of the box.