xcrawl-scraper v1.0.1 — Node.js SDK for Web Scraping

Charles — Thu, 14 May 2026 08:51:26 +0000

I just released xcrawl-scraper v1.0.1 on npm!

Key Features

Scrape any webpage → clean Markdown, JSON, HTML, or text
Search the web → structured results with snippets
Crawl entire sites → built-in sitemap discovery
AI Extraction — describe what you want in plain English, get JSON back
Proxy control — choose exit region (US, JP, DE, GB) or sticky sessions

Quick Start

npm install xcrawl-scraper

const { XCrawlScraper } = require('xcrawl-scraper');
const xcrawl = new XCrawlScraper({ apiKey: 'YOUR_API_KEY' });
const result = await xcrawl.scrapeMarkdown('https://example.com');
console.log(result.data.markdown);

GitHub: https://github.com/yanxvdong123/xcrawl-scraper
npm: https://www.npmjs.com/package/xcrawl-scraper

I Built a Web Scraper API That Handles JS Rendering, CAPTCHAs, and Proxies

Charles — Thu, 14 May 2026 06:29:19 +0000

I got tired of maintaining Puppeteer scripts and proxy lists for every website I needed data from. So I built something better -- a single API that handles JS rendering, proxy rotation, CAPTCHA solving, and AI-powered extraction.

How It Works

1. Scrape any page

\\ash
curl -X POST 'https://run.xcrawl.com/v1/scrape' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": "https://news.ycombinator.com", "output": {"formats": ["markdown"]}}'
\\
Returns clean Markdown. No HTML parsing, no DOM traversal.

2. AI extraction

Describe what you want in English, get structured JSON:
\\ash
curl -X POST 'https://run.xcrawl.com/v1/scrape' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": "https://example.com", "output": {"formats": ["json"]}, "json": {"prompt": "Extract product names and prices"}}'
\\

3. Search the web

\\ash
curl -X POST 'https://run.xcrawl.com/v1/search' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"query": "web scraping best practices", "location": "US"}'
\\

4. Full crawl with proxy control

Choose exit region (US, JP, DE) or sticky session:
\\ash
curl -X POST 'https://run.xcrawl.com/v1/crawl' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": "https://docs.example.com/", "crawler": {"limit": 50, "max_depth": 3}, "proxy": {"location": "JP"}}'
\\

Pricing

Credit-based: basic scrape = 1 credit, search = 2, AI extraction = 5, crawl = 1-5 per page.
Free plan: 1000 credits, no credit card.
Paid plans: Hobby \\/mo -> Starter \\/mo -> Pro \\/mo -> Enterprise \\/mo.

SDK

\\ash
npm install xcrawl-scraper
\\
\\ ypescript
import { XCrawlScraper } from 'xcrawl-scraper';
const x = new XCrawlScraper({ apiKey: process.env.XCRAWL_API_KEY });
const r = await x.scrapeMarkdown('https://example.com');
\\

CLI: \
px xcrawl-scraper scrape https://example.com\

Quick Start

Sign up at dash.xcrawl.com (free, no credit card)
Get your API key
Make your first API call in 30 seconds

Built this because I needed it. Questions? Drop a comment.

DEV Community: Charles

xcrawl-scraper v1.0.1 — Node.js SDK for Web Scraping

Key Features

Quick Start

I Built a Web Scraper API That Handles JS Rendering, CAPTCHAs, and Proxies

How It Works

1. Scrape any page

2. AI extraction

3. Search the web

4. Full crawl with proxy control

Pricing

SDK

Quick Start