From API Key to Production: Setting Up a Web Scraping Pipeline in 10 Minutes

You've got a scraping API key. Now what?

Here's the fastest path from zero to a running scraping pipeline.

Step 1: Get Your API Key

Sign up at dash.xcrawl.com — you get credits for free tier immediately.

Step 2: Install the SDK

npm install xcrawl-scraper

Step 3: Write Your First Pipeline

const { XcrawlScraper } = require('xcrawl-scraper');

const scraper = new XcrawlScraper({ apiKey: process.env.XCRAWL_API_KEY });

async function monitorPrices() {
  const products = ['https://amazon.com/dp/B0EXAMPLE1', 'https://amazon.com/dp/B0EXAMPLE2'];

  const results = await Promise.all(products.map(url => 
    scraper.scrape({ url, js_render: true, proxy: { country: 'US' } })
  ));

  console.log(results);
}

Step 4: Schedule It

# Run every 6 hours via cron
0 */6 * * * node /path/to/monitor.js >> prices.log

Step 5: Scale

Add retry logic, error handling, and CSV export. The API handles rate limits automatically.

---

Get started: dash.xcrawl.com