DEV Community: Simon

How to Scrape Websites with Claude Code (2026 Guide)

Simon — Fri, 17 Apr 2026 22:00:26 +0000

Claude Code can edit files, run shell commands, and write tests — but it cannot fetch live web pages on its own. Connect it to CrawlForge MCP and it gains 20 scraping tools that run straight from your terminal.

npm install -g crawlforge-mcp-server
npx crawlforge-setup  # paste your API key
# Now in Claude Code:
# > Fetch https://news.ycombinator.com and list the top 5 stories

This guide shows you how to scrape websites with Claude Code using CrawlForge MCP, from installation to stealth-mode bypass. Every code block below is runnable.

TL;DR: Claude Code has no network access by default. Install CrawlForge MCP and Claude Code gets 20 scraping tools — fetch_url, extract_content, scrape_with_actions, stealth_mode, and more. Free tier = 1,000 credits, no credit card.

The Problem: Claude Code Cannot Fetch URLs
Prerequisites
Step 1: Install CrawlForge MCP
Step 2: Get Your API Key
Step 3: Register the MCP Server with Claude Code
Step 4: Verify the Connection
Step 5: Your First Scrape
Full Working Example: Scrape a Pricing Page
Advanced: Scrape JavaScript-Rendered Sites
Tool Quick Reference
Troubleshooting
FAQ

The Problem: Claude Code Cannot Fetch URLs

By default, Claude Code has no network access. Ask it to "read this blog post" and it will tell you it cannot open URLs. The built-in WebFetch helper exists in Claude Desktop but is limited, rate-capped, and frequently blocked by Cloudflare, Akamai, and other edge protections.

CrawlForge MCP solves this by exposing 20 scraping tools — fetch_url, extract_content, scrape_structured, stealth_mode, deep_research, and more — as Model Context Protocol tools that Claude Code can call like any other function. For more background on the protocol itself, see the complete guide to MCP web scraping.

+------------+     MCP       +-------------+     HTTPS     +-----------+
| Claude Code| <-----------> | CrawlForge  | <-----------> |  Target   |
| (terminal) |  JSON-RPC     |   Server    |               |   site    |
+------------+   over stdio  +-------------+               +-----------+
                                    |
                                Credits +
                                Stealth +
                                Actions

Prerequisites

Node.js 18+ — check with node --version
Claude Code — install with npm install -g @anthropic-ai/claude-code
A CrawlForge account — free at crawlforge.dev/signup (1,000 credits included, no credit card)

Step 1: Install CrawlForge MCP

npm install -g crawlforge-mcp-server

Verify the install:

crawlforge-mcp-server --version
# crawlforge-mcp-server 3.0.16

Step 2: Get Your API Key

Go to crawlforge.dev/signup and create an account.
Open the dashboard at crawlforge.dev/dashboard/api-keys.
Copy the key — it starts with cf_live_.

Step 3: Register the MCP Server with Claude Code

The fastest path is the setup wizard:

npx crawlforge-setup

It writes the correct entry to ~/.config/claude-code/mcp.json (Linux/macOS) or %APPDATA%\claude-code\mcp.json (Windows).

Prefer manual configuration? Here's the exact JSON

Add this to your Claude Code MCP config file:

{
  "mcpServers": {
    "crawlforge": {
      "command": "crawlforge-mcp-server",
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

Restart Claude Code so it picks up the new server.

Step 4: Verify the Connection

Open Claude Code and run:

/mcp

You should see crawlforge listed as connected with 20 tools available. If not, jump to Troubleshooting.

Step 5: Your First Scrape

Paste this prompt into Claude Code:

Fetch https://news.ycombinator.com using CrawlForge and give me the
top 5 story titles with their URLs as a JSON array.

Claude Code will call fetch_url (1 credit), parse the HTML, and return something like:

[
  { "title": "Show HN: My side project", "url": "https://example.com/post/1" },
  { "title": "Why X is changing Y", "url": "https://example.com/post/2" }
]

That is it. You are scraping.

Full Working Example: Scrape a Pricing Page

Here is a realistic task: extract pricing tiers from a SaaS site. Paste this prompt:

Use scrape_structured to extract pricing from https://crawlforge.dev/pricing.
Return an array of { plan, price, credits, features[] }.

See what Claude Code builds behind the scenes

// What Claude Code sends to CrawlForge via MCP
const response = await fetch('https://crawlforge.dev/api/v1/tools/scrape_structured', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://crawlforge.dev/pricing',
    selectors: {
      plan: '.pricing-card h3',
      price: '.pricing-card .price',
      credits: '.pricing-card .credits',
      features: '.pricing-card ul li',
    },
  }),
});

const data = await response.json();
console.log(data.plan); // ['Free', 'Hobby', 'Professional', 'Business']

Cost: 2 credits. Compare that to running a headless browser locally: zero infrastructure, no Puppeteer debugging, no Cloudflare roulette.

Advanced: Scrape JavaScript-Rendered Sites

Some sites render pricing or product data through client-side React. fetch_url returns the pre-hydration HTML shell and misses the data. Switch to scrape_with_actions (5 credits):

scrape_with_actions — clicks, waits, and scrolls for SPAs

// Prompt Claude Code with the goal, and it generates this call
const payload = {
  url: 'https://app.example.com/dashboard',
  actions: [
    { type: 'wait', selector: '.data-grid', timeout: 5000 },
    { type: 'click', selector: 'button.load-more' },
    { type: 'wait', timeout: 1500 },
  ],
  formats: ['json', 'markdown'],
};

const result = await fetch('https://crawlforge.dev/api/v1/tools/scrape_with_actions', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CRAWLFORGE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(payload),
}).then(r => r.json());

For Cloudflare and Akamai-protected sites, use stealth_mode (also 5 credits). The fingerprint-rotation tradeoffs are covered in the stealth mode deep dive.

Tool Quick Reference

Tool	Credits	When to use
`fetch_url`	1	Static HTML, you will parse yourself
`extract_text`	1	Clean readable text from article pages
`extract_content`	2	Readability-style main content extraction
`scrape_structured`	2	CSS selectors into typed fields
`search_web`	5	You do not know the URL yet
`scrape_with_actions`	5	SPA requires clicks, waits, scrolls
`stealth_mode`	5	Anti-bot systems (Cloudflare, DataDome)
`deep_research`	10	Multi-source research with citations

Full list in the 20-tools overview.

Troubleshooting

MCP server failed to start

Confirm crawlforge-mcp-server is on your PATH:

which crawlforge-mcp-server

If empty, reinstall globally:

npm install -g crawlforge-mcp-server

Unauthorized or 401 errors

Your API key is missing or malformed. It must start with cf_live_. Re-export in your shell and restart Claude Code:

export CRAWLFORGE_API_KEY="cf_live_..."

Insufficient credits

Check usage at crawlforge.dev/dashboard/usage. Free tier = 1,000 credits/month. Upgrade to Hobby ($19/mo) for 25,000.

Tools list is empty when running /mcp

MCP config is not being read. Check the right path for your OS:

macOS: ~/Library/Application Support/claude-code/mcp.json
Linux: ~/.config/claude-code/mcp.json
Windows: %APPDATA%\claude-code\mcp.json

Cloudflare 403 on every fetch

Swap fetch_url for stealth_mode. If you still get blocked, the target uses server-side JA3/JA4 fingerprinting — open a GitHub issue with the URL.

FAQ

Can Claude Code scrape websites without CrawlForge?

Not reliably. Claude Code has no built-in network access, and the WebFetch helper in Claude Desktop is rate-limited and blocked by most anti-bot systems. CrawlForge MCP adds 20 dedicated scraping tools that handle static HTML, JavaScript-rendered pages, and Cloudflare-protected sites.

How much does it cost to scrape with Claude Code?

CrawlForge uses a credit model: basic fetches cost 1 credit, structured extraction 2 credits, search 5, stealth 5, deep research 10. Free accounts get 1,000 credits per month with no credit card. The Hobby plan ($19/mo) includes 25,000 credits.

Why do I get 403 errors when Claude Code fetches certain URLs?

Sites protected by Cloudflare, DataDome, or Akamai block generic HTTP clients via TLS fingerprinting and JavaScript challenges. Switch from fetch_url (1 credit) to stealth_mode (5 credits), which rotates browser fingerprints and solves challenges automatically.

Does CrawlForge MCP work with Claude Code on Windows?

Yes. Install via npm, run npx crawlforge-setup, and the config lands at %APPDATA%\claude-code\mcp.json. Node.js 18+ is the only system requirement. Windows users should run the setup command from PowerShell or Windows Terminal for the cleanest experience.

What is the difference between fetch_url and scrape_with_actions?

fetch_url returns raw HTML via a fast HTTP request (1 credit). scrape_with_actions spins up a headless browser, executes clicks/waits/scrolls, then captures the hydrated DOM (5 credits). Use fetch_url for static pages and scrape_with_actions only when JavaScript rendering is required.

Next Steps

Read the CrawlForge quick start for five copy-paste examples
Browse the getting started docs for full API reference
Compare MCP clients in Claude Desktop vs Claude Code
Evaluate alternatives at Firecrawl alternative

Start Free — 1,000 Credits Included

im about to launch my new saas project next tuesday. kind of excited this will be my first. here is the link in case anyone wants to try it out for free. hope everyone has a great weekend. https://www.crawlforge.dev/

Simon — Fri, 27 Mar 2026 18:18:29 +0000

CrawlForge - Enterprise Web Scraping for AI Applications

18 specialized MCP tools for seamless web scraping and AI-ready data extraction

crawlforge.dev

The Complete Guide to MCP Web Scraping: Everything Developers Need to Know

Simon — Fri, 27 Mar 2026 17:45:08 +0000

The Model Context Protocol (MCP) has fundamentally changed how AI assistants interact with the web. If you've been building with Claude, Cursor, or any MCP-compatible client, you've probably wondered: what's the best way to give your AI agent access to live web data?

This guide covers everything -- from foundational MCP concepts to all 18 CrawlForge tools and how to pick the right one for each job.

TL;DR: MCP is the open standard for connecting AI to tools. CrawlForge is the most comprehensive MCP web scraping server with 18 tools -- 4x more than alternatives. Free tier includes 1,000 credits, no credit card required.

Part 1: Understanding MCP
Part 2: The MCP Web Scraping Ecosystem
Part 3: All 18 Tools Explained
Part 4: Integration Guide
Part 5: Best Practices
Part 6: Getting Started

Part 1: Understanding MCP

What is the Model Context Protocol?

MCP is an open standard developed by Anthropic that lets AI assistants connect to external tools and data sources. Think of it as a universal adapter between AI models and specialized tools.

+--------------+      +----------------+      +---------------+
|   Claude     | <--> |  MCP Server    | <--> |  External     |
|  (AI Model)  |      |  (CrawlForge)  |      |  Resources    |
+--------------+      +----------------+      +---------------+
                             ^
                      MCP Protocol
                   (JSON-RPC over stdio)

Why MCP Matters for Web Scraping

Before MCP, getting real-time web data into AI was painful:

Approach	Problems
Training data	Outdated, knowledge cutoff
RAG (Retrieval)	Limited to indexed documents
Function calling	Requires custom implementation
Browser plugins	Inconsistent, security concerns

MCP solves these with:

Standardized interface -- One protocol for all tools
Real-time data -- Fresh information from any source
Tool composability -- Combine multiple tools seamlessly
Security model -- Controlled access to external resources

How MCP Works

MCP uses a client-server architecture with JSON-RPC:

Step 1: Server Discovery

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["crawlforge-mcp-server"]
    }
  }
}

Step 2: Tool Registration -- the server advertises its capabilities

{
  "tools": [
    {
      "name": "fetch_url",
      "description": "Fetch content from a URL",
      "inputSchema": {
        "type": "object",
        "properties": {
          "url": { "type": "string", "format": "uri" }
        },
        "required": ["url"]
      }
    }
  ]
}

Step 3: Tool Invocation -- the AI calls a tool

{
  "method": "tools/call",
  "params": {
    "name": "fetch_url",
    "arguments": { "url": "https://example.com" }
  }
}

Step 4: Response -- structured data comes back

{
  "result": {
    "content": "<html>...",
    "status": 200,
    "headers": {}
  }
}

Part 2: The MCP Web Scraping Ecosystem

Several MCP servers provide web scraping capabilities:

Server	Tools	Focus
CrawlForge	18	Comprehensive scraping, research, stealth
Firecrawl	~5	Basic scraping and crawling
Browser MCP	~3	Browser automation
Fetch MCP	1	Simple HTTP requests

CrawlForge: ████████████████████ 18 tools
Firecrawl:  █████                 5 tools
Browser:    ███                   3 tools
Fetch:      █                     1 tool

Part 3: CrawlForge's 18 Tools Explained

Basic Scraping (1-2 credits)

1. fetch_url (1 credit) -- Fetch raw HTML from any URL

The foundation of web scraping. Always try this first.

// Usage: "Fetch https://example.com"
// Returns:
{
  html: "<html>...",
  status: 200,
  headers: { /* ... */ },
  timing: { total: 523 }
}

When to use: Starting point for any scraping task.

2. extract_text (1 credit) -- Clean text extraction

Removes HTML tags, scripts, and styles. Returns readable text with word count and reading time.

When to use: Blog posts, articles, documentation where you need readable text.

3. extract_links (1 credit) -- Discover all links on a page

{
  url: "https://example.com",
  filter_external: true  // Internal links only
}
// Returns: { links: [{ href: "/about", text: "About Us" }, ...], total: 45 }

When to use: Site exploration, finding pages to scrape, building sitemaps.

4. extract_metadata (1 credit) -- SEO metadata extraction

Pulls title, description, Open Graph, JSON-LD structured data.

When to use: SEO analysis, content previews, structured data extraction.

Structured Extraction (2-3 credits)

5. scrape_structured (2 credits) -- CSS selector-based extraction

{
  url: "https://example.com/products",
  selectors: {
    title: "h1.product-title",
    price: "span.price",
    description: ".product-description"
  }
}
// Returns: { data: { title: "Product Name", price: "$99.99", description: "..." } }

When to use: E-commerce scraping, structured data, known page layouts.

6. extract_content (2 credits) -- Intelligent article extraction

Like Readability -- returns title, author, published date, clean content, images, reading time.

When to use: News articles, blog posts, editorial content.

7. map_site (2 credits) -- Site structure discovery

{
  url: "https://example.com",
  max_urls: 1000,
  include_sitemap: true
}
// Returns: pages[], structure tree

When to use: Site audits, crawl planning, content discovery.

8. analyze_content (3 credits) -- NLP analysis

Language detection, sentiment scoring, topic extraction, named entity recognition, readability analysis.

When to use: Content analysis, sentiment tracking, topic extraction.

Advanced Scraping (4-5 credits)

9. process_document (2 credits) -- PDF and document handling

{
  source: "https://example.com/report.pdf",
  sourceType: "pdf_url"
}
// Returns: { text: "...", pages: 15, metadata: { author, created } }

When to use: Research papers, reports, documentation PDFs.

10. summarize_content (4 credits) -- AI-powered summarization

Returns concise summary, key points, and word reduction percentage.

When to use: Long documents, research synthesis, content digests.

11. crawl_deep (4 credits) -- Multi-page crawling

{
  url: "https://example.com",
  max_depth: 3,
  max_pages: 100,
  include_patterns: ["/blog/*"],
  exclude_patterns: ["/admin/*"]
}

When to use: Full site scraping, content aggregation, archiving.

12. batch_scrape (5 credits) -- Parallel scraping of multiple URLs

Scrape up to 50 URLs in parallel with concurrency control.

When to use: Multiple known URLs, competitor monitoring, price tracking.

13. scrape_with_actions (5 credits) -- Browser automation

{
  url: "https://example.com/app",
  actions: [
    { type: "wait", selector: ".content" },
    { type: "click", selector: "#load-more" },
    { type: "wait", timeout: 2000 },
    { type: "scroll", selector: "body" },
    { type: "screenshot" }
  ]
}

When to use: SPAs, infinite scroll, dynamic content, login-required pages.

14. search_web (5 credits) -- Google search integration

{
  query: "web scraping best practices 2026",
  limit: 10,
  site: "github.com"  // Optional site filter
}

When to use: Discovery, finding sources, research starting point.

Specialized Tools (2-10 credits)

15. stealth_mode (5 credits) -- Anti-detection bypass

Hides WebDriver, randomizes fingerprints, simulates human behavior. Configurable stealth levels.

When to use: Protected sites, Cloudflare bypass, anti-bot evasion.

16. track_changes (3 credits) -- Content monitoring

Set a baseline, then detect changes with before/after diffs and significance scoring.

When to use: Price monitoring, competitor tracking, content updates.

17. localization (2 credits) -- Geo-targeted scraping

Set country code and language to get region-specific content and pricing.

When to use: Regional pricing, localized content, geo-restricted data.

18. deep_research (10 credits) -- Multi-source research

{
  topic: "quantum computing commercialization",
  maxUrls: 50,
  enableSourceVerification: true,
  enableConflictDetection: true
}
// Returns: { synthesis, sources[], conflicts[], citations[] }

When to use: Research projects, due diligence, market analysis.

Part 4: Integration Guide

Claude Code Setup

# 1. Install CrawlForge MCP server
npm install -g crawlforge-mcp-server

# 2. Run setup wizard
npx crawlforge-setup

# 3. Add to Claude Code
claude
> /mcp add crawlforge npx crawlforge-mcp-server

# 4. Verify
> /mcp list
# Should show: crawlforge (18 tools)

Claude Desktop Setup

Edit your config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["crawlforge-mcp-server"],
      "env": {
        "CRAWLFORGE_API_KEY": "cf_live_your_key_here"
      }
    }
  }
}

Custom Application (MCP SDK)

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

const transport = new StdioClientTransport({
  command: "npx",
  args: ["crawlforge-mcp-server"],
  env: { CRAWLFORGE_API_KEY: process.env.CRAWLFORGE_API_KEY }
});

const client = new Client({ name: "my-app", version: "1.0.0" });
await client.connect(transport);

// List available tools
const tools = await client.listTools();
console.log(`Available tools: ${tools.tools.length}`);

// Call a tool
const result = await client.callTool({
  name: "fetch_url",
  arguments: { url: "https://example.com" }
});

Part 5: Best Practices

Credit Optimization

Pick the cheapest tool that gets the job done:

Goal	Expensive	Efficient
Check if page exists	`deep_research` (10)	`fetch_url` (1)
Get article text	`scrape_with_actions` (5)	`extract_content` (2)
Find competitor URLs	`search_web` x 10 (50)	`extract_links` (1)
Scrape 20 product pages	`fetch_url` x 20 (20)	`batch_scrape` (5)

Error Handling

try {
  const result = await fetchUrl(url);
  if (result.status >= 400) {
    // Try with stealth mode for blocked requests
    return await stealthMode(url);
  }
  return result;
} catch (error) {
  await sleep(retryDelay * attempt);
  return retry(url, attempt + 1);
}

Rate Limiting

Respect target sites:

// Good: Reasonable delays
for (const url of urls) {
  await scrape(url);
  await sleep(1000 + Math.random() * 2000); // 1-3s delay
}

// Better: Use batch_scrape with built-in rate limiting
await batchScrape(urls, { delayBetweenRequests: 1500 });

Caching

Don't scrape the same URL twice:

const cache = new Map<string, ScrapedContent>();

async function smartScrape(url: string) {
  if (cache.has(url)) return cache.get(url);

  const result = await fetchUrl(url);
  cache.set(url, result);
  return result;
}

Part 6: Getting Started

Free Tier -- 1,000 one-time trial credits. All 18 tools available. No credit card required.

npm install -g crawlforge-mcp-server
npx crawlforge-setup

What 1,000 Credits Gets You

Use Case	Tool	Credits	Capacity
Basic scraping	`fetch_url`	1	1,000 pages
Article extraction	`extract_content`	2	500 articles
Site mapping	`map_site`	2	500 sites
Batch jobs	`batch_scrape`	5	200 batches
Research projects	`deep_research`	10	100 topics

Key Takeaways

MCP is the standard -- All major AI assistants support it
CrawlForge leads with 18 tools -- 4x more than alternatives
Start simple -- Use fetch_url (1 credit) before reaching for advanced tools
Combine tools -- Chain operations for powerful workflows
Be ethical -- Respect robots.txt and rate limits

More from CrawlForge:

Start Free -- 1,000 Credits Included