Giving Your AI Coding Assistant Live Web Access with an MCP Server

XCrawl — Sat, 17 Oct 2026 20:01:34 +0000

Large language models know a lot, but they don't know what changed on a website this morning. If you use Claude Code, Claude Desktop, Cursor or another MCP client, you can close that gap by connecting a web data server through the Model Context Protocol (MCP). This post explains what that gives you, how to set it up in a few minutes, and how to use it without wasting tokens.

What MCP adds

MCP is an open protocol that lets an AI client call external tools. A web scraping MCP server exposes tools such as "scrape this URL" or "search the web" that the assistant can call on its own when a task needs fresh information: reading the latest docs for a library, checking a changelog, comparing pricing pages, or collecting examples from a site.

Which tools you get

As an example, the XCrawl MCP server (npm package xcrawl-mcp, for the XCrawl web data API) exposes these tools:

xcrawl_scrape: fetch one page and return Markdown, HTML, links or a summary
xcrawl_search: run a web search and return structured results
xcrawl_map: list the URLs a domain exposes
xcrawl_crawl and xcrawl_check_crawl_status: crawl a site section as a background job and collect results
xcrawl_key_status: check your API key and remaining credits

Option 1: the hosted endpoint

The simplest setup uses the hosted MCP URL, so nothing runs on your machine. In Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "xcrawl": {
      "url": "https://mcp.xcrawl.com/YOUR_API_KEY/mcp"
    }
  }
}

In Claude Code you can add the same endpoint from the terminal:

claude mcp add --transport http xcrawl https://mcp.xcrawl.com/YOUR_API_KEY/mcp

The server also accepts the key in an Authorization: Bearer header if you'd rather keep it out of the URL.

Option 2: run it locally over stdio

If you prefer a local process (Node.js 18+):

XCRAWL_API_KEY=YOUR_API_KEY npx -y xcrawl-mcp

or register it with Claude Code:

claude mcp add xcrawl -e XCRAWL_API_KEY=YOUR_API_KEY -- npx -y xcrawl-mcp

Prompts that work well

"Read the migration guide at and list the breaking changes that affect our code."
"Search for the current rate limits of and cite the source pages."
"Map the URLs under docs.example.com/guides and summarize what each guide covers."

Keeping it efficient

Ask for Markdown, not HTML: it is far smaller and easier for the model to read.
Map before you crawl: listing URLs first lets you pick only the pages you need.
Prefer a few targeted scrapes over a full crawl when you just need one answer.
Keep sources: ask the assistant to include the URLs it read so you can verify claims.
Be a good citizen: respect each site's robots.txt and terms, and don't hammer small sites.

Wrapping up

Connecting a web data MCP server turns your assistant from "trained up to some date" into something that can check the live web when it matters. Setup is a single config entry, and new XCrawl accounts start with 1,000 free credits, which is plenty to try the workflow on your own projects. Docs are at https://docs.xcrawl.com/.

DEV Community: XCrawl

Giving Your AI Coding Assistant Live Web Access with an MCP Server