I Built a Web Scraper API That Handles JS Rendering, CAPTCHAs, and Proxies

#webscraping #webdev #javascript

I got tired of maintaining Puppeteer scripts and proxy lists for every website I needed data from. So I built something better -- a single API that handles JS rendering, proxy rotation, CAPTCHA solving, and AI-powered extraction.

How It Works

1. Scrape any page

\\ash
curl -X POST 'https://run.xcrawl.com/v1/scrape' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": "https://news.ycombinator.com", "output": {"formats": ["markdown"]}}'
\\
Returns clean Markdown. No HTML parsing, no DOM traversal.

2. AI extraction

Describe what you want in English, get structured JSON:
\\ash
curl -X POST 'https://run.xcrawl.com/v1/scrape' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": "https://example.com", "output": {"formats": ["json"]}, "json": {"prompt": "Extract product names and prices"}}'
\\

3. Search the web

\\ash
curl -X POST 'https://run.xcrawl.com/v1/search' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"query": "web scraping best practices", "location": "US"}'
\\

4. Full crawl with proxy control

Choose exit region (US, JP, DE) or sticky session:
\\ash
curl -X POST 'https://run.xcrawl.com/v1/crawl' \
-H 'Authorization: Bearer ***' \
-H 'Content-Type: application/json' \
-d '{"url": "https://docs.example.com/", "crawler": {"limit": 50, "max_depth": 3}, "proxy": {"location": "JP"}}'
\\

Pricing

Credit-based: basic scrape = 1 credit, search = 2, AI extraction = 5, crawl = 1-5 per page.
Free plan: 1000 credits, no credit card.
Paid plans: Hobby \\/mo -> Starter \\/mo -> Pro \\/mo -> Enterprise \\/mo.

SDK

\\ash
npm install xcrawl-scraper
\\
\\ ypescript
import { XCrawlScraper } from 'xcrawl-scraper';
const x = new XCrawlScraper({ apiKey: process.env.XCRAWL_API_KEY });
const r = await x.scrapeMarkdown('https://example.com');
\\