DEV Community

Zenrows profile image Praise James
Praise James for Zenrows

Posted on Originally published at zenrows.com

How to scrape protected sites in Cursor with Zenrows MCP

#mcp #ai #agents #zenrows

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/scrape-protected-sites-cursor

This guide shows you how to add Zenrows MCP to Cursor so your agent can reach protected, JavaScript-rendered, and recently updated pages that Cursor's built-in web access can't. You need Node.js v18+, a Zenrows API key, and Cursor installed.

Setup takes one JSON config file and a restart. Three workflows follow to verify it works on real sites.

Before you start

  • Node.js v18 or later. Run node --version to confirm
  • A Zenrows API key from app.zenrows.com
  • Cursor installed

Why Cursor's built-in web access falls short

Three categories break the cached-snapshot approach Cursor uses by default.

JavaScript-rendered pages. Cursor fetches the HTML shell and misses the data that loads at runtime.

Bot-protected sites. Pages behind Cloudflare, login walls, or anti-bot systems block the request and return a cached fragment. No error is surfaced.

Recently updated pages. Documentation, pricing, or API references changed in the last few hours. The cached copy may be days old.

The failure is usually silent. Your agent returns something, just not what the page actually contains.

Step 1: Create mcp.json

OS Path
macOS / Linux ~/.cursor/mcp.json
Windows %APPDATA%\Cursor\mcp.json

On macOS / Linux:

mkdir -p ~/.cursor && cat > ~/.cursor/mcp.json << 'EOF'
{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "your_api_key_here"
      }
    }
  }
}
EOF
Enter fullscreen mode Exit fullscreen mode

On Windows, open %USERPROFILE%\.cursor\ in File Explorer, create mcp.json, and paste the same JSON.

Step 2: Validate the JSON

Run this before opening Cursor. A trailing comma or missing bracket stops the config loading silently:

node -e "JSON.parse(require('fs').readFileSync(require('os').homedir()+'/.cursor/mcp.json','utf8')); console.log('JSON valid')"
Enter fullscreen mode Exit fullscreen mode

JSON valid means you're good. Any error here is a syntax problem in the file.

Step 3: Confirm the connection

Open Cursor → gear icon (bottom left) → search "MCP" → Tools & MCPs → Home MCP Servers. Look for zenrows with a green dot and 37 tools enabled.

On macOS, use ⌘Q to fully quit rather than closing the window — the MCP server process reads your API key on spawn, so closing the window isn't enough.

Workflow 1: Fetch live data from a protected site

Paste this into Cursor's agent:

Use the Zenrows MCP scrape tool to fetch https://www.scrapingcourse.com/antibot-challenge
and return the content as clean Markdown.

Do not use your built-in web browsing. Only use the Zenrows scrape tool.
Enter fullscreen mode Exit fullscreen mode

Cursor's built-in browsing returns the challenge interstitial for this URL. Zenrows clears the anti-bot challenge and returns the page behind it as clean Markdown.

Workflow 2: Extract structured data from a JavaScript-rendered page 

Use the Zenrows MCP scrape tool to fetch https://www.scrapingcourse.com/javascript-rendering
and extract a structured JSON object containing the first 5 products with their name and price.
Do not use your built-in web browsing.
Enter fullscreen mode Exit fullscreen mode

The products load dynamically. A plain HTTP request returns an empty grid. Zenrows renders the page first, then the agent parses the products out of the #product-grid container.

Expected output shape:

[
  { "name": "Product Name", "price": "$XX.XX" },
  ...
]
Enter fullscreen mode Exit fullscreen mode

Workflow 3: Generate TypeScript from live protected data

This one combines two fetch levels with code generation. Category listings on Home Depot carry product names and prices. SKU and model number only exist on each individual product page, so the agent has to fetch both levels and combine them.

Use the Zenrows MCP scrape tool to fetch
https://www.homedepot.com/b/Tools-Woodworking-Tools/N-5yc1vZc2gv?catStyle=ShowProducts

and return the first 5 products as structured JSON with name and price. Then fetch each
product's individual page and add its SKU and model number.
Set proxy country to US. Do not use your built-in web browsing.

Then write a TypeScript function that fetches and displays product details for a given
product name.
Enter fullscreen mode Exit fullscreen mode

proxy_country: US routes the request through a US IP so the retailer returns its normal catalogue regardless of where you run Cursor.

The agent returns structured JSON across both page levels:

{
  "products": [
    {
      "name": "Gorilla 4 fl. oz. Wood Glue",
      "price": "$3.98",
      "sku": "1003827526",
      "model_number": "62020"
    },
    {
      "name": "DEWALT 20V MAX XR Cordless Brushless Fixed Base Compact Router (Tool Only)",
      "price": "$249.00",
      "sku": "1004095707",
      "model_number": "DCW600B"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Then it generates a TypeScript function with this signature:

export async function fetchAndDisplayProductDetails(
  productName: string,
  apiKey = process.env.ZENROWS_API_KEY ?? ""
): Promise<HomeDepotProductDetails | null>
Enter fullscreen mode Exit fullscreen mode

Run it with:

export ZENROWS_API_KEY="your_api_key_here"
npm run build
npm run homedepot -- "Gorilla 4 fl. oz. Wood Glue"
Enter fullscreen mode Exit fullscreen mode

Results are saved to data/homedepot-products.json.

Zenrows vs Firecrawl in Cursor

Both work in Cursor. Use Firecrawl for public pages where speed and simplicity matter. Use Zenrows when the page is protected, JavaScript-rendered, or part of a high-volume workflow.

Scenario Recommended Reason
Public blog or open docs Firecrawl Fast Markdown from the open web
Multi-page crawl on public sites Firecrawl Built-in crawl and map workflows
Protected site behind Cloudflare Zenrows JS rendering and premium proxies handle challenge pages
JavaScript-rendered page with embedded data Zenrows Renders the full page before extraction
Recently updated docs or pricing Zenrows Live fetch, not a cached copy
High-volume recurring data workflow Zenrows Batch, retry logic, and observability built in
Agent needs structured JSON output Zenrows Returns HTML, Markdown, JSON, screenshots, or plain text

Full comparison: Zenrows vs Firecrawl

Debugging

Zenrows MCP isn't appearing after I added the config
Restart Cursor fully (⌘Q on macOS). Validate your JSON with the command in Step 2.

401 error in the agent response
ZENROWS_API_KEY in mcp.json is wrong or still the placeholder. Check it against a live key in your Zenrows account, save, quit Cursor fully, and reopen.

EACCES error on macOS

sudo chown -R $(id -u):$(id -g) ~/.npm
Enter fullscreen mode Exit fullscreen mode

Then restart Cursor.

Prefer the hosted MCP server over npx?
Use https://mcp.zenrows.com/mcp with your API key as a Bearer token in Cursor's remote MCP configuration.

What's next

Top comments (0)