DEV Community

johann
johann

Posted on

How to Build an Anti-Bot Web Scraping MCP Server for Claude & Cursor with Clean Markdown Extraction in Python

#webscraping #python #ai #mcp

How to Build an Anti-Bot Web Scraping MCP Server for Claude & Cursor with Clean Markdown Extraction in Python

Autonomous AI assistants in Claude Desktop, Cursor, Windsurf, and Antigravity are capable of sophisticated research, technical synthesis, and data analysis. However, as soon as you ask an AI model to read a live web page or scrape documentation, it encounters three immediate barriers:

  1. Anti-Bot WAFs & HTTP 403 Blocks: Default Python HTTP libraries send identifiable signatures that trigger Cloudflare and Datadome bot challenges immediately.
  2. Context Window Pollution: A standard modern web page is 200KB of raw HTML, containing 90% boilerplate (tracking scripts, cookie consent popups, navigation bars, SVG sprites, and ads). Feeding this raw DOM tree to an LLM burns tokens and induces hallucinations.
  3. Empty JavaScript SPAs: Modern web applications built on React, Next.js, or Vue render blank containers without server-side hydration.

The modern architectural solution is to deploy a Model Context Protocol (MCP) server that sits between the AI assistant and the web, rotating realistic browser fingerprints and delivering clean, semantic Markdown.

In this tutorial, we will build CrawlEngine MCP in Python to give AI agents reliable, token-efficient web-reading capabilities.

Architecture: The Resilient Extraction Pipeline 

[ AI Agent: Claude / Cursor / Antigravity ]
                      │  JSON-RPC 2.0 (stdio)
                      ▼
           [ CrawlEngine MCP Server ]
                      │
        [ Browser Fingerprint Rotator ]
        ├─ Realistic Sec-Ch-Ua & Platform Headers
        ├─ Rotating User-Agents (macOS, Windows, Linux)
        └─ Organic Referer Spoofing
                      │
           [ Proxy Pool Manager ]
           └─ HTTP / SOCKS5 Round-Robin Rotation
                      │
             [ HTTP / DOM Engine ]
                      │
         [ Semantic Markdown Extractor ]
        ├─ Strips scripts, navbars, ads, cookie overlays
        ├─ Converts tables into structured Markdown
        └─ Up to 85% Token Reduction
                      │
                      ▼
[ Clean, Token-Efficient Markdown Output to Agent ]
Enter fullscreen mode Exit fullscreen mode

1. Realistic Browser Fingerprinting

Web Application Firewalls look beyond simple User-Agent strings; they analyze Client Hints (Sec-Ch-Ua headers) to ensure the declared platform matches the browser engine:

import random

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
]

def get_realistic_headers(target_url: str):
    ua = random.choice(USER_AGENTS)
    is_mac = "Macintosh" in ua
    platform = '"macOS"' if is_mac else '"Windows"'

    return {
        "User-Agent": ua,
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Sec-Ch-Ua": '"Chromium";v="128", "Google Chrome";v="128"',
        "Sec-Ch-Ua-Platform": platform,
        "Referer": "https://www.google.com/"
    }
Enter fullscreen mode Exit fullscreen mode

2. Converting DOM to Clean Markdown (Token Optimization)

To prevent wasting context window tokens, we decompose non-content tags (<script>, <style>, <nav>, <footer>) and strip elements containing cookie and banner classes:

from bs4 import BeautifulSoup
import re

NOISE_TAGS = ["script", "style", "noscript", "nav", "footer", "aside", "iframe"]
NOISE_CLASSES = re.compile(r"(cookie|banner|advert|popup|modal|newsletter)", re.IGNORECASE)

def clean_html_to_markdown(html_content: str) -> str:
    soup = BeautifulSoup(html_content, "html.parser")

    # Decompose noise
    for tag in soup(NOISE_TAGS):
        tag.decompose()

    for el in soup.find_all(attrs={"class": NOISE_CLASSES}):
        if el.name not in ["main", "article", "body"]:
            el.decompose()

    main = soup.find("main") or soup.find("article") or soup.body or soup

    lines = []
    for tag in main.find_all(["h1", "h2", "h3", "p", "li"]):
        text = tag.get_text(strip=True)
        if not text:
            continue
        if tag.name == "h1":
            lines.append(f"\n# {text}\n")
        elif tag.name == "h2":
            lines.append(f"\n## {text}\n")
        elif tag.name == "p":
            lines.append(f"\n{text}\n")
        elif tag.name == "li":
            lines.append(f"- {text}")

    return "\n".join(lines).strip()
Enter fullscreen mode Exit fullscreen mode

3. Configuring Claude Desktop & Cursor

Connect the server in your claude_desktop_config.json:

{
  "mcpServers": {
    "crawlengine": {
      "command": "python",
      "args": ["-m", "crawlengine.server"],
      "cwd": "/path/to/crawlengine-mcp"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Now you can instruct Claude:

"Scrape https://news.ycombinator.com, extract the top 10 articles, and summarize the key technical trends."

Enterprise Edition: Proxy Pool Rotation & Structured Data Extraction

When scraping behind corporate firewalls or parsing e-commerce pricing tables, the Enterprise Pro edition includes:

  • Proxy Pool Manager: Automated round-robin rotation of HTTP/SOCKS5 proxies with automatic dead-proxy ejection.
  • Structured Schema Extractor: Parses embedded JSON-LD schemas and HTML <table> elements into structured JSON objects.
  • Production Docker Container: Multi-stage image ready to deploy as an internal microservice.

Downloads and Packages:

  • Lite Version (Open Source): Semantic Markdown conversion and browser fingerprint rotation.
  • Enterprise Pro Edition ($14.99): Proxy pool rotation, structured JSON-LD/table parsing, Docker Compose, and automated test suite.

👉 Get CrawlEngine Pro on Gumroad ($14.99)

Top comments (0)