Scrape E-commerce Sites at the Lowest Layer That Works

You point a scraper at a product page, get a 200 response, and still extract nothing useful. The HTML has a root div, a few script tags, maybe a loading skeleton. Prices, variants, reviews, stock status, and seller data arrive later through JavaScript. If you retry with a headless browser, you may get the data, but you also inherit browser fingerprints, CAPTCHA risk, slower jobs, and more moving parts.

The useful question is not “which scraper should I use?” It is “which layer has the data?”

Start below the browser

Most e-commerce pages are not built from static HTML anymore. The frontend renders a page shell, then calls backend endpoints for search results, product details, reviews, availability, recommendations, and pricing.

Open DevTools, go to the Network tab, filter by Fetch/XHR, then reload a product page. You will usually see requests returning JSON. That is the first place to look.

A rough workflow:

1. Load the page manually in a browser.
2. Filter network requests to XHR or Fetch.
3. Search response bodies for a known product title, SKU, price, or review text.
4. Copy the request as cURL.
5. Remove headers one by one until you find the minimum request that still works.

For some sites, the request works directly:

curl 'https://example.com/api/products/12345' \
  -H 'accept: application/json' \
  -H 'user-agent: Mozilla/5.0'

For others, it fails with a 403, returns { "error": "Unauthorized" }, or sends an empty result unless you include a session cookie, geo header, store ID, or CSRF token. That tells you the browser is doing more than displaying the page. It is creating state that the API expects.

For marketplaces where those internal endpoints change often, Wire exposes catalogued network-layer actions for product search, product detail, reviews, and seller data without asking you to maintain DOM selectors.

Shopify is the easy case people forget

Before launching Playwright at a Shopify store, check /products.json.

Many Shopify stores expose their catalog as paginated JSON:

curl 'https://storename.myshopify.com/products.json?limit=250'

The response includes product titles, handles, variants, prices, image URLs, and inventory policy fields. You can paginate with page or cursor-style parameters depending on the storefront configuration.

A simple collector might look like this:

const base = 'https://storename.myshopify.com/products.json?limit=250';

async function fetchAllProducts() {
  let page = 1;
  const products = [];

  while (true) {
    const res = await fetch(`${base}&page=${page}`);

    if (!res.ok) {
      throw new Error(`Shopify request failed: ${res.status}`);
    }

    const data = await res.json();

    if (!data.products || data.products.length === 0) break;

    products.push(...data.products);
    page += 1;
  }

  return products;
}

This is faster and less fragile than parsing collection pages. It also avoids a common failure mode: your CSS selector works today, then the theme changes and your scraper silently starts returning null prices.

There are limits. Some stores block /products.json with Cloudflare. Some hide inventory quantities. Some large catalogs do not expose everything through the public endpoint. But you should still test this path first because it costs almost nothing to verify.

Use rendering when data exists only after JavaScript runs

Some sites do not expose a convenient public JSON endpoint, or they require JavaScript to build tokens before calling internal APIs. In that case, render the page and extract from the DOM or captured responses.

Playwright gives you both options:

import { chromium } from 'playwright';

const browser = await chromium.launch();
const page = await browser.newPage();

page.on('response', async (response) => {
  const url = response.url();

  if (url.includes('/api/') && response.headers()['content-type']?.includes('application/json')) {
    try {
      const json = await response.json();
      console.log('API response:', url, json);
    } catch {
      // Not all JSON-looking responses parse cleanly.
    }
  }
});

await page.goto('https://example.com/product/12345', {
  waitUntil: 'networkidle'
});

const price = await page.locator('[data-testid="price"]').textContent();
console.log({ price });

await browser.close();

This works, but it is slower. A direct JSON request might finish in a few hundred milliseconds. A browser render often takes several seconds, and at volume you must manage memory, retries, proxy routing, timeouts, and browser crashes.

Typical symptoms at this layer:

• TimeoutError: page.goto: Timeout 30000ms exceeded
• CAPTCHA pages returned with status 200
• product grids that render only the first page because scrolling was not simulated
• prices changing because the site inferred the wrong region
• selectors returning null after a frontend deploy

Rendering is a fallback, not a default.

Use a full browser only for interactive state

Some workflows genuinely need a browser:

• login-protected prices
• wishlists
• cart totals
• shipping estimates
• loyalty pricing
• checkout availability
• store-specific inventory after selecting a location

Trying to fake these with raw HTTP can turn into a pile of brittle cookie handling and hidden token extraction. If the workflow requires clicks, navigation, persisted sessions, or user-specific state, use a browser and make that cost explicit.

The tradeoff is simple: browser automation reaches more flows, but each job costs more time and infrastructure. You also need session storage so you do not log in on every run. Repeated logins are noisy and often trigger account protection.

A practical decision tree

I usually choose the layer in this order:

1. Public JSON endpoint, such as Shopify /products.json or a WooCommerce catalog API.
2. Internal XHR or Fetch endpoint discovered through DevTools.
3. Rendered HTML with a headless browser.
4. Full browser automation with saved sessions for login or cart flows.

On catalogued marketplaces like Amazon, eBay, Walmart, and Flipkart, Wire fits the second category: network-layer extraction where the useful data sits behind frontend API calls rather than stable public endpoints.

The important part is to avoid starting at layer four. It works in demos, then becomes expensive and flaky when you scale from ten URLs to ten thousand.

For your next scraper, pick one target page and spend ten minutes in the Network tab before writing extraction code. If you find a clean JSON response, build around that. If you do not, then move up one layer and document exactly why the lower layer failed.

Comments

[magia c'c]

The store ID and region examples are a reason to record context alongside every price, whichever layer returns it. For a repeatable BrowserAct Cloud Bot, I'd make those inputs explicit and export them with currency and capture time; otherwise two successful runs may describe different offers.