How to Scrape JS-Rendered E-Commerce Pages Without Getting Blocked (2026 Guide)

#webscraping #javascript #webdev #tutorial

The Problem

E-commerce sites like Amazon, Walmart, and Target have moved to heavy JavaScript rendering. Traditional HTTP clients (curl, requests, fetch) return empty HTML shells - no product data, no prices, no reviews.

You need a headless browser to render the JS. But headless browsers are slow, expensive, and easy to detect.

Common Anti-Bot Detection Signals

Modern anti-bot systems check for:

Missing User-Agent or non-standard headers
Headless Chrome flags (navigator.webdriver === true)
Abnormal mouse movement patterns
Request frequency (too fast = bot)
IP reputation (data center IPs are often flagged)

Enter XCrawl

XCrawl is a web scraping proxy API that handles all of this automatically:

JS Rendering - Every request goes through a real browser engine
Proxy Rotation - Residential and datacenter proxies, auto-rotated
CAPTCHA Bypass - Automatic solving for common CAPTCHAs
Sticky Sessions - Keep the same IP for multi-page crawls

Quick Demo: Scrape a Product Page

Here's how you'd scrape an Amazon product page with the xcrawl-scraper Node.js SDK:

import { XCrawl } from 'xcrawl-scraper'

const client = new XCrawl({
  apiKey: 'your-api-key'
})

const result = await client.scrapeMarkdown({
  url: 'https://www.amazon.com/dp/B0B1N1L78J',
  proxyLocation: 'us',        // Use US residential IP
  waitForSelector: '#productTitle'  // Wait for content to load
})

console.log(result.content)

Why This Matters

Without these features, you'd need to:

Spin up a Puppeteer/Playwright instance
Manage a proxy pool yourself
Handle CAPTCHAs manually
Deal with IP blocks and rate limits

XCrawl bundles all of this into a single API call. One endpoint, one SDK, done.

Pricing

Free: 1000 credits (enough for testing)
Hobby: $8/mo for 50K credits
Starter: $49/mo for 500K credits
Pro: $199/mo for 3M credits

Try It Yourself

Sign up at dash.xcrawl.com (free tier available)
Grab your API key
Run the code above

Full documentation and SDK: github.com/yanxvdong123/xcrawl-scraper

I built this SDK and API. Questions? Drop a comment below.

DEV Community