CSS selectors against a rendered product page are a trap. They look fine until marketing ships a redesign on a Tuesday and your div.price__amount--v2 returns null for every SKU. Then you're spinning up a headless browser just to read a number that was in the HTTP response the whole time.
Here's the thing I wish someone had told me three storefronts ago: the product data is almost always already JSON, sitting right there in the page. You don't parse it out of the DOM. You pluck it out of a script tag or an XHR. I've built scrapers for a couple dozen retailers now, and I reach for selectors maybe once a year.
Three places to look, best first.
JSON-LD, the one retailers hand you on purpose
Retailers want Google's rich results, so they embed schema.org/Product markup. That's a <script type="application/ld+json"> block with price, currency, availability, sometimes a GTIN — a documented shape you can rely on across completely unrelated sites.
import { gotScraping } from 'got-scraping';
async function fromJsonLd(url) {
const res = await gotScraping({ url });
const blocks = [...res.body.matchAll(
/<script type="application\/ld\+json"[^>]*>([\s\S]*?)<\/script>/g,
)];
for (const [, json] of blocks) {
const data = JSON.parse(json);
const product = [].concat(data, data['@graph'] ?? [])
.find((n) => n && n['@type'] === 'Product');
if (product) {
return {
title: product.name,
price: product.offers?.price ?? product.offers?.[0]?.price,
currency: product.offers?.priceCurrency,
availability: product.offers?.availability, // schema.org/InStock etc.
gtin: product.gtin13 ?? product.gtin,
};
}
}
return null;
}
No browser. No selectors. The shape is a public standard, so the same parser works on a shoe store and a hardware store.
Where it falls down: JSON-LD is the marketing-facing summary, not the truth. I've hit pages where the <script> block cheerfully says InStock while the actual buy button is greyed out, because the stock check happens client-side after hydration. So JSON-LD is my first call, never my only one.
Embedded app state — __NEXT_DATA__, __NUXT__, window.__INITIAL_STATE__
Anything built on Next, Nuxt, or a similar framework hydrates from a JSON blob baked into the HTML. This is usually the good stuff. Full variant matrices, per-size stock, image sets, seller info — everything the page needs to render, which is a lot more than JSON-LD bothers to expose.
const m = res.body.match(/<script id="__NEXT_DATA__"[^>]*>([\s\S]*?)<\/script>/);
const state = JSON.parse(m[1]);
// then dig to state.props.pageProps.product — path varies per site
The path down to the product object is site-specific and annoying to find the first time. Open the blob, JSON.stringify a few candidate nodes, eyeball which one has the price. Once you've mapped it for a retailer it stays put, and it's the richest source you'll get short of the internal API.
The internal mobile JSON API
When there's no JSON-LD and no embedded state — or when the embedded state is deliberately thin — open the network tab and watch what the page fetches. Most SPAs pull the product from an XHR that returns clean JSON, something along the lines of /api/product/{id}. Hit that directly and you skip rendering entirely; you're reading the exact payload the site's own frontend trusts.
This is the approach I lean on hardest for the SHEIN scraper. SHEIN's desktop pages are heavy and defensive, but the site's internal mobile JSON API returns the product already structured — title, price, the full variant list, images — no HTML parsing at any step. I recently added a keyword-search mode on top of it, so you can hand the actor a search term, let it walk the same mobile API for matching product IDs, then run each ID through the detail path. Discovery and extraction, both off the JSON API, no browser in the loop.
One catch that cost me an afternoon the first time: a lot of these internal endpoints return a 403 to a plain fetch. It looks like an IP ban, so you go buy proxies, and it's still 403. It's not your IP — it's your TLS and header fingerprint. Node's default client doesn't handshake like a real browser, and these endpoints notice. got-scraping exists for exactly this; it mimics a browser's TLS fingerprint and header order, and the same request that was getting bounced starts returning 200 with the JSON you wanted.
// plain fetch -> 403, no matter how many proxies you throw at it
// got-scraping -> 200, because the fingerprint reads as a real browser
const res = await gotScraping({ url: internalApiUrl, responseType: 'json' });
Fetching is the easy 20%. Normalization is the other 80%
Getting one store's JSON is a lazy afternoon. The part that actually eats your time is that every retailer names everything differently. Price is price, or current_price, or salePrice.amount. Availability is a boolean here, a schema.org URL there, a raw stock integer somewhere else. Variants come as a flat array on one site and a nested option matrix on the next. A multi-store dataset is worthless unless every product lands in one fixed schema:
{
title, price, currency, availability, // normalized to in_stock | out_of_stock
variants: [{ name, price, available, sku }],
gtin, images, url
}
That mapping layer — a small adapter per retailer, all feeding one output shape — is where every hour of maintenance goes. It's also the whole difference between "I scraped a store once" and "I have a product feed I can trust on Monday morning."
When you don't want to babysit the adapters
Owning a tuned adapter per retailer, and re-tuning each one when the site shifts a field, is a real recurring cost. If you'd rather POST a product URL and get normalized JSON back across 20-plus retailers, that's the exact chore I'm building Cartpie to eat — one schema across every store. And if SHEIN is your only target, the SHEIN product scraper on Apify does the single-store version.
Scraping one or two stores yourself? The three techniques above cover you with no browser anywhere. JSON-LD first, embedded state when you need the variant matrix, internal API when the site forces your hand.
What's the ugliest product schema you've had to flatten? Mine was a marketplace that base64-encoded the real price and buried it three levels deep, presumably so competitors wouldn't do exactly what I was doing.
Top comments (2)
This is so true! I've had more robust results targeting
application/ld+jsonblocksAround 80% of my dataset of many millions of e-commerce product pages comes from ld+json. The last 20% is enormous effort though 😅 I have spent years working on unblocking, scraping, extracting, normalising product data. If you want me to share some more data or posts on some specific topics please let me know