DEV Community

XCrawl
XCrawl

Posted on Originally published at xcrawl.hashnode.dev

Giving Your AI Coding Assistant Live Web Access with an MCP Server

#mcp #ai #webscraping #tutorial

Large language models know a lot, but they don't know what changed on a website this morning. If you use Claude Code, Claude Desktop, Cursor or another MCP client, you can close that gap by connecting a web data server through the Model Context Protocol (MCP). This post explains what that gives you, how to set it up in a few minutes, and how to use it without wasting tokens.

What MCP adds

MCP is an open protocol that lets an AI client call external tools. A web scraping MCP server exposes tools such as "scrape this URL" or "search the web" that the assistant can call on its own when a task needs fresh information: reading the latest docs for a library, checking a changelog, comparing pricing pages, or collecting examples from a site.

Which tools you get

As an example, the XCrawl MCP server (npm package xcrawl-mcp, for the XCrawl web data API) exposes these tools:

  • xcrawl_scrape: fetch one page and return Markdown, HTML, links or a summary
  • xcrawl_search: run a web search and return structured results
  • xcrawl_map: list the URLs a domain exposes
  • xcrawl_crawl and xcrawl_check_crawl_status: crawl a site section as a background job and collect results
  • xcrawl_key_status: check your API key and remaining credits

Option 1: the hosted endpoint

The simplest setup uses the hosted MCP URL, so nothing runs on your machine. In Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "xcrawl": {
      "url": "https://mcp.xcrawl.com/YOUR_API_KEY/mcp"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

In Claude Code you can add the same endpoint from the terminal:

claude mcp add --transport http xcrawl https://mcp.xcrawl.com/YOUR_API_KEY/mcp
Enter fullscreen mode Exit fullscreen mode

The server also accepts the key in an Authorization: Bearer header if you'd rather keep it out of the URL.

Option 2: run it locally over stdio

If you prefer a local process (Node.js 18+):

XCRAWL_API_KEY=YOUR_API_KEY npx -y xcrawl-mcp
Enter fullscreen mode Exit fullscreen mode

or register it with Claude Code:

claude mcp add xcrawl -e XCRAWL_API_KEY=YOUR_API_KEY -- npx -y xcrawl-mcp
Enter fullscreen mode Exit fullscreen mode

Prompts that work well

  • "Read the migration guide at and list the breaking changes that affect our code."
  • "Search for the current rate limits of and cite the source pages."
  • "Map the URLs under docs.example.com/guides and summarize what each guide covers."

Keeping it efficient

  • Ask for Markdown, not HTML: it is far smaller and easier for the model to read.
  • Map before you crawl: listing URLs first lets you pick only the pages you need.
  • Prefer a few targeted scrapes over a full crawl when you just need one answer.
  • Keep sources: ask the assistant to include the URLs it read so you can verify claims.
  • Be a good citizen: respect each site's robots.txt and terms, and don't hammer small sites.

Wrapping up

Connecting a web data MCP server turns your assistant from "trained up to some date" into something that can check the live web when it matters. Setup is a single config entry, and new XCrawl accounts start with 1,000 free credits, which is plenty to try the workflow on your own projects. Docs are at https://docs.xcrawl.com/.

Top comments (0)